← All commands

chimera swe-bench-compare

Honest A/B over two SWE-bench Verified-Mini reports on the SAME instance ids. Reads the official evaluation reports (``resolved_ids`` or a per-instance map) for a free model alone vs the same model driven by Chimera, projects both onto the shared instance list (a missing id counts as unresolved), and prints the delta + 95% CI. This is the second standard scoreboard for the weak-model-lift thesis; the pass/fail comes from SWE-bench's tests, never self-reported.