Evaluation Platform API

The Evaluation Platform’s callable surface is the eval harness: test cases in evals.json, an iteration runner, and an aggregator that produces a benchmark verdict.

Eval case format

Each suite lives in .claude/skills/<skill>/evals/:

.claude/skills/<skill>/
├── SKILL.md
└── evals/
    ├── evals.json     # test cases: prompt + assertions
    └── files/         # input fixtures

An eval case is a prompt plus assertions — concrete, checkable statements about what a correct output must contain (figures traceable to filings, live citations, expected file artifacts). Fixtures in files/ pin the inputs so runs are reproducible.

Running an iteration

git submodule update --init --recursive   # one-time: pull the skill sources

Then ask the orchestrator: “run evals for <skill>” — it follows the scripts/run_eval.md runbook:

Fresh subagent per case, no carry-over context.
Inputs: the test prompt, fixtures, and an output directory under evals/<skill>/iteration-N/.
Timing and token usage captured immediately at task completion.
The previous version becomes the old_skill baseline for the next iteration.

Aggregation and verdict

python scripts/aggregate.py   # evals/<skill>/iteration-N/ -> benchmark.json

benchmark.json is the per-iteration verdict: assertion pass rates against the baseline, plus cost/latency per case. A regression against the previous iteration blocks promotion to production — the same gate the Research Agent and Strategy Pipeline ship through.

Pipeline under test

The quant research suites cover the full pipeline as separate, individually benchmarked stages:

paper-search -> paper-extract -> paper-replicate -> strategy-translate

Each stage has its own evals/ suite, so a regression is attributed to the stage that introduced it.

Get started

Research Agent

Strategy Pipeline