chimera hierarchy-bench

Paired A/B: single-agent (all docs inline) vs the hierarchy (one worker per doc). Calls real models. Both arms run on the SAME model so the comparison isolates the ORCHESTRATION (minimal-context scoping + budgets + contracts), not model strength. Quality = paired McNemar/Wilson (the only place "significant" appears); tokens = measured totals per arm, with no significance claim on cost. `--multistep` switches to the companion suite where the token crossover lives: a single agent re-sends every document on every turn (cost grows with turns), while scoped workers pay each doc ~once — and prices the measured cache reduction via the caching model.

Options

--model, -mstr
Mid/worker model — BOTH arms use it, to isolate orchestration. Defaults to the tier ladder's mid.
--top-modelstr
Top model for synthesis. Defaults to --model (same family keeps the isolation).
--tasksstrdefault: ''
Comma-separated task ids to filter (default: all 10 synthetic tasks).
--max-workersintdefault: 4
Max concurrent workers in the hierarchy arm.
--outstr
Write the JSON summary to this path.
--multistepboolean
Run the MULTI-STEP suite instead (single growing context vs per-step scoped workers, over large docs) — the regime where the hierarchy actually saves tokens. Also reports a caching-aware dollar reduction.