The continuous-evolution benchmark, and every other ruler as a subcommand.
Four-arm bench: weak-only vs mid-only vs cascade vs fusion. Calls real models. Published criterion (stated up front): cascade >= mid-only pass rate at materially lower tokens-per-pass. The number reported is whatever is measured.
--tasksstrdefault: 'hard'Task suite: hard | demo.
Report the honest A/B delta (+95% CI) between two benchmark result files. Feed it the pass/fail from two runs on the SAME task IDs (e.g. a terminal-bench free-model baseline vs the same model driven by Chimera). Prints each arm's Wilson-bounded pass rate, the delta, its Newcombe CI, and whether the difference is significant. This is the number that proves (or doesn't) that the scaffolding lifts a weak model. With --paired, the two lists are treated as *aligned pairs* (each index is one task replayed from an identical forked checkpoint), and the tighter McNemar/Wilson interval is reported — the payoff of running both arms from the same forked state.
BASELINEstrrequiredJSON file of the baseline arm's per-task pass/fail (list of bools, or {task: bool}).
TREATMENTstrrequiredJSON file of the treatment arm's per-task pass/fail.
--baseline-namestrdefault: 'baseline'Label for the baseline arm.
--treatment-namestrdefault: 'chimera'Label for the treatment arm.
--pairedbooleanPaired (McNemar) test: item i in both files is the SAME task replayed from an identical forked state — a tighter CI.
Did runs carrying more context do worse? Measured on THIS machine's own logs. Answers with "not enough data" until the pre-registered floors are met — see `bench/context_curve/PREREGISTRATION.md`, which fixed those floors before any data existed.
--tracesstrPath to traces.jsonl (default: CHIMERA_HOME).
--runsstrPath to runs.jsonl (default: CHIMERA_HOME).
--jsonbooleanPrint the raw result instead of a table.
Stress-test continuous-evolution degradation: naive vs guarded. Requires a key.
--lengthintdefault: 12Number of chained steps.
--model, -mstrOverride the model slug.
--retriesintdefault: 2Verify-or-revert retries per step (guarded).
A/B the fusion engine: full vs selective (tokens + accuracy). Calls real models.
--tasksstrdefault: 'hard'Task suite: hard | demo.
Paired A/B: single-agent (all docs inline) vs the hierarchy (one worker per doc). Calls real models. Both arms run on the SAME model so the comparison isolates the ORCHESTRATION (minimal-context scoping + budgets + contracts), not model strength. Quality = paired McNemar/Wilson (the only place "significant" appears); tokens = measured totals per arm, with no significance claim on cost. `--multistep` switches to the companion suite where the token crossover lives: a single agent re-sends every document on every turn (cost grows with turns), while scoped workers pay each doc ~once — and prices the measured cache reduction via the caching model.
--model, -mstrMid/worker model — BOTH arms use it, to isolate orchestration. Defaults to the tier ladder's mid.
--top-modelstrTop model for synthesis. Defaults to --model (same family keeps the isolation).
--tasksstrdefault: ''Comma-separated task ids to filter (default: all 10 synthetic tasks).
--max-workersintdefault: 4Max concurrent workers in the hierarchy arm.
--outstrWrite the JSON summary to this path.
--multistepbooleanRun the MULTI-STEP suite instead (single growing context vs per-step scoped workers, over large docs) — the regime where the hierarchy actually saves tokens. Also reports a caching-aware dollar reduction.
Measure recall@k as memory grows — lexical vs paraphrase. Default (keyword search, no key needed) surfaces the honest ceiling: exact-token recall holds at scale, but paraphrase recall collapses. Pass ``--semantic`` to re-run with the embedding recall path and watch the paraphrase column lift — that delta is the whole point of M11b.
--sizesstrdefault: '50,200,1000'Comma-separated memory sizes to sweep.
--semanticbooleanUse embedding recall (needs an embeddings key) to measure the lift.
Ablate the memory-poisoning defenses: what reaches a LATER run's prompt, and unmarked. No key needed, nothing leaves the machine. `redteam` measures one run — content arrives untrusted, the harmful call is refused, and the whole picture ends with the process. This measures the other shape: run A stores what it "learned" from a poisoned page, run B asks an unrelated question days later, and recall hands the planted fact to the model. The headline is what arrives **unmarked**, not what is blocked. A poisoned fact carrying its origin is one the model was warned about; an unlabelled one is indistinguishable from something the agent verified itself. Each of the three layers (taint / gate / label) is switched off in turn, because a single number would be compatible with any of them doing nothing. See `bench/memory_poison/PREREGISTRATION.md` for the thresholds, fixed before the first run.
This command takes no options.
PROBE best-arm identification with a cheap-proxy control variate (M18-5). "Which model/config is best?" where each expensive reward (a real grade) is paired with a cheap proxy (a weak judge) of unknown correlation. PROBE uses the proxy as a control variate so the estimate needs FEWER expensive draws the better the proxy correlates — and stays unbiased when the proxy is useless. Prints each arm's adjusted mean ± interval, the winner, and — if not yet confident — the arm to sample next. Feed it recorded (proxy, reward) observations from a bench.
DATAstrJSON: {"arm": [[proxy, reward-or-null], ...], ...}. Omit when using --from-log.
--from-logstrRead observations from a ProbeLog JSONL (e.g. <home>/probe.jsonl written by `solve --probe-log`).
--deltafloatdefault: 0.1Confidence level (smaller = stricter).
--min-rewardintdefault: 2Expensive rewards required per arm before deciding.
Grade an answer against an authorable rubric — weighted criteria with a required-criterion veto. Produces a per-criterion breakdown, a single weighted score, and a pass/fail verdict. A required criterion that falls below the gate vetoes the outcome regardless of the weighted score.
--rubricstrrequiredJSON rubric: {criteria:[{text,weight,required}], pass_threshold, required_gate}.
--taskstrrequiredThe task the answer is for.
--answerstrThe answer text (or use --answer-file).
--answer-filestrRead the answer from this file.
--modelstrModel slug for the grader.
State-based bench: grade the final workspace state + count harmful side effects. Unlike the text benches, this measures what the agent DID (files it changed), and flags mutations outside each task's allowed set. Uses real models + file tools.
--workspace, -wstrdefault: '.sandbox-bench'Dir to run sandboxed tasks in.
--model, -mstrOverride the model slug.
--max-stepsintdefault: 8Max tool-calling steps per task.
Measure tool-schema token cost, full vs compacted (advertise-time). No model calls.
--openapistrPath or URL to an OpenAPI spec to include (its tools are verbose).
--demobooleanInclude a couple of synthetic verbose tools to show the effect.
--model, -mstrTokenizer model (default: your default).
A/B reasoning with vs without injected TRS skill cards. Calls real models.
--tasksstrdefault: 'hard'Task suite: hard | big | demo. 'big' = 24 traps for a tighter paired CI.
--kintdefault: 1How many cards to retrieve per task.
--min-overlapintdefault: 2Relevance gate: inject a card only on >= N shared query terms (0=off).
--max-linesintdefault: 3Render budget: max lines per injected card.
--use-storebooleanBench your own learned cards (skills.json) instead of the demo set.
Honest A/B over two SWE-bench Verified-Mini reports on the SAME instance ids. Reads the official evaluation reports (``resolved_ids`` or a per-instance map) for a free model alone vs the same model driven by Chimera, projects both onto the shared instance list (a missing id counts as unresolved), and prints the delta + 95% CI. This is the second standard scoreboard for the weak-model-lift thesis; the pass/fail comes from SWE-bench's tests, never self-reported.
BASELINEstrrequiredSWE-bench evaluation report JSON for the model-only arm.
TREATMENTstrrequiredSWE-bench evaluation report JSON for the model+Chimera arm.
--instancesstrrequiredJSONL of the instances both arms ran (fixes the id set).
Promote a learned change only if it helps its tuned slice AND doesn't regress a holdout. Guards against *negative transfer* — a GEPA prompt / ACE delta / distilled skill that raises the pass rate on the tasks it was tuned against but REGRESSES on other tasks sharing the capability. Feed the tuned slice's paired pass/fail (baseline vs candidate) and, ideally, a disjoint same-capability holdout's; the verdict is PROMOTE / BLOCK with the paired evidence (exit 1 on BLOCK, for CI). Without a holdout it promotes on the tuned gain alone but flags that transfer was NOT measured.
TUNED_BASELINEstrrequiredJSON pass/fail of the baseline on the TUNED slice (list of bools, or {task: bool}).
TUNED_TREATMENTstrrequiredJSON pass/fail of the candidate on the TUNED slice (aligned, same order).
--holdout-baselinestrJSON pass/fail of the baseline on a DISJOINT same-capability holdout.
--holdout-treatmentstrJSON pass/fail of the candidate on the holdout (aligned).
--require-significantbooleanRequire the tuned gain's paired CI to exclude 0, not just Δ>0.
--tolfloatdefault: 0.0Max tolerated pass-rate drop on the holdout before promotion is blocked.