Evidence

Every number this project publishes, including the ones where it lost.

These figures are read from the snapshots the product generates and stamps at each release. Nothing on this page is typed by hand, which is also why none of it can quietly age.

Documenting version 0.42.0

internal Docker-free suite (local_lift), pytest-graded — NOT SWE-bench/Terminal-Bench

Significant

Measured on our own Docker-free suite, graded by pytest — one model, small self-contained Python tasks. This is not SWE-bench, and it does not generalise to real repositories.

Baseline: 48%
With Chimera: 71%
Difference: +23pp
95% confidence interval: [+12.6%, +28.6%]
Sample: n = 100
Model: openrouter/mistralai/mistral-small-3.2-24b-instruct

A cheap weak model + Chimera's retry loop vs the cheap model alone, on a pre-registered n=100 suite (design + tasks committed before any model call). SIGNIFICANT: the paired 95% CI excludes zero. The lift is 28 tasks the loop RECOVERED (raw fail → verified pass) against 5 regressions. This is the RE-VERIFIED run: an earlier run of the same suite (9% → 15%) was graded with a test file the agent under test could edit — and on re-run it did edit one — so grading was hardened to restore the pristine test, and the lift replicated LARGER, not smaller. The superseded run is kept unedited beside the erratum in bench/local_lift/RESULTS.md. One model, one seed/task, small self-contained Python tasks — NOT SWE-bench, does not generalise to real repos. One run, no re-roll.

Source: bench/local_lift/_reverify_n100/paired.json

SWE-bench Verified (41-instance django easy slice, out-of-sample replication)

Not significant

A deliberately easy, single-repo slice of SWE-bench Verified. This is not a SWE-bench Verified score — a real one needs the full 500 instances. The delta is not significant on its own.

Baseline: 34.1%
With Chimera: 43.9%
Difference: +9.8pp
95% confidence interval: [-3.5%, +16.7%]
Sample: n = 41
Model: openrouter/deepseek/deepseek-chat-v3.1

NOT a SWE-bench Verified score — a deliberately easy, single-repo slice; a real score needs the full 500. Graded only by the official swebench harness in Docker, never self-reported. This is the OUT-OF-SAMPLE replication: 41 instances never used by the earlier run, nothing else changed. The delta is NOT significant on its own (CI includes 0); pooling it with the earlier 19 gives +11.7% [+0.8%, +16.4%], which IS significant but mixes seen with unseen data and was pre-registered as secondary. DECOMPOSITION (4th run, middle arm restored on the same 41): scaffold alone +4.9% over baseline, the diff-gate a further +4.9% — BOTH components contribute in roughly equal halves, neither significant alone. That CONTRADICTED our own registered prediction that the scaffold would carry most of it, and withdrew an earlier reading that the gate was not what produced the gain; the retraction is in RESULTS.md. The tidy additivity is NOT claimed as a measured 50/50 split — each comparison rests on 5-6 discordant pairs. All three arms edit at the SAME rate (27-28 patches of 41); what climbs is precision, 50% -> 59% -> 67%. An earlier run of the same design at a starved step budget scored an exact 0.0pp and is published unchanged.

Source: bench/swe_bench/RESULTS.md

Terminal-Bench (terminal-bench-core 0.1.1)

Not significant

The scaffold did not help here, and the run is published as measured. Both arms sit near the floor and the difference is not significant.

Baseline: 7.5%
With Chimera: 2.5%
Difference: -5pp
95% confidence interval: [-5%, +1.6%]
Sample: n = 40
Model: openrouter/deepseek/deepseek-chat-v3.1

Single-attempt, N=40, same model both arms. The scaffold did NOT lift here — deepseek-v3.1 is already competent, not the weak 'goldilocks' regime where scaffolding helps; both arms sit near the floor and the difference is not significant (CI includes 0). Published as measured; the point estimate is dominated by run-to-run variance at this floor.

Source: bench/terminal_bench/RESULTS.md

Surface coverage

Internal surface coverage — how much of the codebase is proven by real tests. It is not a statement about the product, which is alpha.

37 of 37 criteria proven

fusion5/5
evolution8/8
governance7/7
memory4/4
benchmarks5/5
resilience6/6
interop2/2

Retracted claim

The agent extracts a skill from every verified success and reads them back on later runs. Whether that accumulation makes it measurably better at new tasks was tested across seven pre-registered runs. The sixth found a positive; the seventh, with more power, cut it to nothing.

Retracted. Seven pre-registered runs; the one positive result did not survive a higher-powered replication, so the claim was withdrawn rather than kept.