Evidence

Every number this project publishes, including the ones where it lost.

These figures are read from the snapshots the product generates and stamps at each release. Nothing on this page is typed by hand, which is also why none of it can quietly age.

Documenting version 0.42.0

internal Docker-free suite (local_lift), pytest-graded — NOT SWE-bench/Terminal-Bench

Significant

测于我们自己的免 Docker 测试集，由 pytest 判分——单一模型、小型自包含的 Python 任务。这不是 SWE-bench，也不能推广到真实仓库。

Baseline: 48%
With Chimera: 71%
Difference: +23pp
95% confidence interval: [+12.6%, +28.6%]
Sample: n = 100
Model: openrouter/mistralai/mistral-small-3.2-24b-instruct

A cheap weak model + Chimera's retry loop vs the cheap model alone, on a pre-registered n=100 suite (design + tasks committed before any model call). SIGNIFICANT: the paired 95% CI excludes zero. The lift is 28 tasks the loop RECOVERED (raw fail → verified pass) against 5 regressions. This is the RE-VERIFIED run: an earlier run of the same suite (9% → 15%) was graded with a test file the agent under test could edit — and on re-run it did edit one — so grading was hardened to restore the pristine test, and the lift replicated LARGER, not smaller. The superseded run is kept unedited beside the erratum in bench/local_lift/RESULTS.md. One model, one seed/task, small self-contained Python tasks — NOT SWE-bench, does not generalise to real repos. One run, no re-roll.

Source: bench/local_lift/_reverify_n100/paired.json

SWE-bench Verified (41-instance django easy slice, out-of-sample replication)

Not significant

SWE-bench Verified 中一个刻意挑选的、来自单一仓库的简单切片。这不是 SWE-bench Verified 的分数——真正的分数需要完整的 500 个实例。这个差值单独来看并不显著。

Baseline: 34.1%
With Chimera: 43.9%
Difference: +9.8pp
95% confidence interval: [-3.5%, +16.7%]
Sample: n = 41
Model: openrouter/deepseek/deepseek-chat-v3.1

NOT a SWE-bench Verified score — a deliberately easy, single-repo slice; a real score needs the full 500. Graded only by the official swebench harness in Docker, never self-reported. This is the OUT-OF-SAMPLE replication: 41 instances never used by the earlier run, nothing else changed. The delta is NOT significant on its own (CI includes 0); pooling it with the earlier 19 gives +11.7% [+0.8%, +16.4%], which IS significant but mixes seen with unseen data and was pre-registered as secondary. DECOMPOSITION (4th run, middle arm restored on the same 41): scaffold alone +4.9% over baseline, the diff-gate a further +4.9% — BOTH components contribute in roughly equal halves, neither significant alone. That CONTRADICTED our own registered prediction that the scaffold would carry most of it, and withdrew an earlier reading that the gate was not what produced the gain; the retraction is in RESULTS.md. The tidy additivity is NOT claimed as a measured 50/50 split — each comparison rests on 5-6 discordant pairs. All three arms edit at the SAME rate (27-28 patches of 41); what climbs is precision, 50% -> 59% -> 67%. An earlier run of the same design at a starved step budget scored an exact 0.0pp and is published unchanged.

Source: bench/swe_bench/RESULTS.md

Terminal-Bench (terminal-bench-core 0.1.1)

Not significant

支架在这里没有带来提升，该次运行按测得的样子公开。两组都贴近地板，差异并不显著。

Baseline: 7.5%
With Chimera: 2.5%
Difference: -5pp
95% confidence interval: [-5%, +1.6%]
Sample: n = 40
Model: openrouter/deepseek/deepseek-chat-v3.1

Single-attempt, N=40, same model both arms. The scaffold did NOT lift here — deepseek-v3.1 is already competent, not the weak 'goldilocks' regime where scaffolding helps; both arms sit near the floor and the difference is not significant (CI includes 0). Published as measured; the point estimate is dominated by run-to-run variance at this floor.

Source: bench/terminal_bench/RESULTS.md

Surface coverage

内部表面覆盖率——代码中有多少被真实测试证明。它不是对产品的判断，产品处于 alpha 阶段。

37 of 37 criteria proven

fusion5/5
evolution8/8
governance7/7
memory4/4
benchmarks5/5
resilience6/6
interop2/2

Retracted claim

The agent extracts a skill from every verified success and reads them back on later runs. Whether that accumulation makes it measurably better at new tasks was tested across seven pre-registered runs. The sixth found a positive; the seventh, with more power, cut it to nothing.

已撤回。七次预注册的运行；唯一那次阳性结果没能通过一次功效更高的复现，因此该结论被撤回而不是保留。