Przejdź do treści

← All commands

chimera measure

Run the rulers this project measures itself with.

Subcommands

chimera measure rag

Recall@k of each retriever over a real folder — lexical, and vector when an embedder is set. This is the measurement `chimera/rag/__init__.py` names when it says the retriever's existence is not a claim that it helps. That sentence pointed at a module you could not run: `rag_bench` had no caller outside its own test and was not exported from `chimera.eval`. Without `--semantic` no embedder is passed, so the vector and hybrid figures come back as None rather than zero — an embedder that was never called did not fail, and printing 0.0 invites the wrong conclusion. With it, the run that `bench/rag/RESULTS.md` reports is reproducible from the CLI rather than from a script somebody has to write. It costs an embedding pass over the corpus: about two cents for this repository's 3,459 chunks and 400 probes, and the figure it produces belongs to the embedder that produced it — vector spaces do not convert between models.

Arguments

  • ROOTpathrequired

    Folder to index and probe.

Options

  • --kintdefault: 10

    Retrieve this many chunks per probe.

  • --max-probesintdefault: 200

    Cap the probe count; each one is a query.

  • --semanticboolean

    Measure the vector and hybrid arms too. Costs money.

chimera measure reranker

Leave-one-out AUC of the success reranker — does it discriminate, or is it noise? `chimera/evolution/reranker.py` says to measure with this BEFORE putting the reranker in a hot path. It was prose pointing at an unreachable module. AUC of 0.5 is a coin flip. A reranker at 0.5 is not a weak reranker, it is not a reranker.

Arguments

  • CORPUSpathrequired

    JSONL of {query, text, success} records.

Options

  • --kintdefault: 5

    Rank cut-off for the leave-one-out scoring.