Evidence

Every number this project publishes, including the ones where it lost.

These figures are read from the snapshots the product generates and stamps at each release. Nothing on this page is typed by hand, which is also why none of it can quietly age.

Documenting version 0.42.0

internal Docker-free suite (local_lift), pytest-graded — NOT SWE-bench/Terminal-Bench

Significant

Misurato sulla nostra suite senza Docker, valutata da pytest — un solo modello, piccoli compiti Python autonomi. Questo non è SWE-bench, e non si generalizza a repository reali.

Baseline: 48%
With Chimera: 71%
Difference: +23pp
95% confidence interval: [+12.6%, +28.6%]
Sample: n = 100
Model: openrouter/mistralai/mistral-small-3.2-24b-instruct

A cheap weak model + Chimera's retry loop vs the cheap model alone, on a pre-registered n=100 suite (design + tasks committed before any model call). SIGNIFICANT: the paired 95% CI excludes zero. The lift is 28 tasks the loop RECOVERED (raw fail → verified pass) against 5 regressions. This is the RE-VERIFIED run: an earlier run of the same suite (9% → 15%) was graded with a test file the agent under test could edit — and on re-run it did edit one — so grading was hardened to restore the pristine test, and the lift replicated LARGER, not smaller. The superseded run is kept unedited beside the erratum in bench/local_lift/RESULTS.md. One model, one seed/task, small self-contained Python tasks — NOT SWE-bench, does not generalise to real repos. One run, no re-roll.

Source: bench/local_lift/_reverify_n100/paired.json

SWE-bench Verified (41-instance django easy slice, out-of-sample replication)

Not significant

Una fetta deliberatamente facile e su un solo repository di SWE-bench Verified. Questo non è un punteggio SWE-bench Verified — uno vero richiede tutte le 500 istanze. La differenza non è significativa da sola.

Baseline: 34.1%
With Chimera: 43.9%
Difference: +9.8pp
95% confidence interval: [-3.5%, +16.7%]
Sample: n = 41
Model: openrouter/deepseek/deepseek-chat-v3.1

NOT a SWE-bench Verified score — a deliberately easy, single-repo slice; a real score needs the full 500. Graded only by the official swebench harness in Docker, never self-reported. This is the OUT-OF-SAMPLE replication: 41 instances never used by the earlier run, nothing else changed. The delta is NOT significant on its own (CI includes 0); pooling it with the earlier 19 gives +11.7% [+0.8%, +16.4%], which IS significant but mixes seen with unseen data and was pre-registered as secondary. DECOMPOSITION (4th run, middle arm restored on the same 41): scaffold alone +4.9% over baseline, the diff-gate a further +4.9% — BOTH components contribute in roughly equal halves, neither significant alone. That CONTRADICTED our own registered prediction that the scaffold would carry most of it, and withdrew an earlier reading that the gate was not what produced the gain; the retraction is in RESULTS.md. The tidy additivity is NOT claimed as a measured 50/50 split — each comparison rests on 5-6 discordant pairs. All three arms edit at the SAME rate (27-28 patches of 41); what climbs is precision, 50% -> 59% -> 67%. An earlier run of the same design at a starved step budget scored an exact 0.0pp and is published unchanged.

Source: bench/swe_bench/RESULTS.md

Terminal-Bench (terminal-bench-core 0.1.1)

Not significant

L'impalcatura qui non ha aiutato, e l'esecuzione è pubblicata così come è stata misurata. Entrambi i bracci sono vicini al pavimento e la differenza non è significativa.

Baseline: 7.5%
With Chimera: 2.5%
Difference: -5pp
95% confidence interval: [-5%, +1.6%]
Sample: n = 40
Model: openrouter/deepseek/deepseek-chat-v3.1

Single-attempt, N=40, same model both arms. The scaffold did NOT lift here — deepseek-v3.1 is already competent, not the weak 'goldilocks' regime where scaffolding helps; both arms sit near the floor and the difference is not significant (CI includes 0). Published as measured; the point estimate is dominated by run-to-run variance at this floor.

Source: bench/terminal_bench/RESULTS.md

Surface coverage

Copertura interna di superficie — quanta parte del codice è provata da test reali. Non è un'affermazione sul prodotto, che è in alpha.

37 of 37 criteria proven

fusion5/5
evolution8/8
governance7/7
memory4/4
benchmarks5/5
resilience6/6
interop2/2

Retracted claim

The agent extracts a skill from every verified success and reads them back on later runs. Whether that accumulation makes it measurably better at new tasks was tested across seven pre-registered runs. The sixth found a positive; the seventh, with more power, cut it to nothing.

Ritrattato. Sette esecuzioni preregistrate; l'unico risultato positivo non ha superato una replica con più potenza, quindi l'affermazione è stata ritirata invece che mantenuta.