The Evaluation Platform is how the other three services stay reliable in production. It instruments the failure modes LLM research actually exhibits — hallucinated figures, stale data, and broken citations — and defines verifiers for portfolio construction and alpha-research workflows, so regressions are caught before deployment rather than after.

​
Failure modes under test

​
Verifier suites

Suites are prioritized by blast radius:
  1. Beta-audit — is a “beta” strategy just repackaged factor exposure?
  2. Portfolio construction — do allocations respect constraints and survive transaction costs?
  3. Fundamental analysis — are figures, dates, and citations in a thesis traceable to filings?
  4. Alpha research (wq-alpha-research) — does a candidate signal actually rank-predict out of sample?
  5. Quant paper research — does a replication match the paper’s methodology and headline result?

​
The iteration loop

Evals are structured as prompt + assertion test cases, run against baselines:
1

Add test cases

Each skill or workflow owns an evals.json of prompts with concrete, machine-checkable assertions, plus input fixtures.
2

Run against baselines

Fresh evaluation runs with no carry-over context compare the current version against a with/without or old/new baseline — you see the delta, not just the level.
3

Grade with concrete evidence

Assertions are graded on quotes, file sizes, and AST checks — never “looks fine.”
4

Benchmark and gate

Results aggregate into a per-iteration benchmark.json. A regression against the previous baseline blocks production deployment.

Eval API

The eval case format, harness commands, and benchmark artifacts.