chimera bench-compare

Report the honest A/B delta (+95% CI) between two benchmark result files. Feed it the pass/fail from two runs on the SAME task IDs (e.g. a terminal-bench free-model baseline vs the same model driven by Chimera). Prints each arm's Wilson-bounded pass rate, the delta, its Newcombe CI, and whether the difference is significant. This is the number that proves (or doesn't) that the scaffolding lifts a weak model. With --paired, the two lists are treated as *aligned pairs* (each index is one task replayed from an identical forked checkpoint), and the tighter McNemar/Wilson interval is reported — the payoff of running both arms from the same forked state.

Arguments

BASELINEstrrequired
JSON file of the baseline arm's per-task pass/fail (list of bools, or {task: bool}).
TREATMENTstrrequired
JSON file of the treatment arm's per-task pass/fail.

Options

--baseline-namestrdefault: 'baseline'
Label for the baseline arm.
--treatment-namestrdefault: 'chimera'
Label for the treatment arm.
--pairedboolean
Paired (McNemar) test: item i in both files is the SAME task replayed from an identical forked state — a tighter CI.