Przejdź do treści
Chimera
Pobierz
Polski

← All commands

chimera measure

Run the rulers this project measures itself with.

Subcommands

chimera measure rag

Recall@k of each retriever over a real folder — lexical, and vector when an embedder is set. This is the measurement `chimera/rag/__init__.py` names when it says the retriever's existence is not a claim that it helps. That sentence pointed at a module you could not run: `rag_bench` had no caller outside its own test and was not exported from `chimera.eval`. No embedder is passed, so the vector and hybrid figures come back as None rather than zero — an embedder that was never called did not fail, and printing 0.0 invites the wrong conclusion.

Arguments

  • ROOTpathrequired

    Folder to index and probe.

Options

  • --kintdefault: 10

    Retrieve this many chunks per probe.

  • --max-probesintdefault: 200

    Cap the probe count; each one is a query.

chimera measure reranker

Leave-one-out AUC of the success reranker — does it discriminate, or is it noise? `chimera/evolution/reranker.py` says to measure with this BEFORE putting the reranker in a hot path. It was prose pointing at an unreachable module. AUC of 0.5 is a coin flip. A reranker at 0.5 is not a weak reranker, it is not a reranker.

Arguments

  • CORPUSpathrequired

    JSONL of {query, text, success} records.

Options

  • --kintdefault: 5

    Rank cut-off for the leave-one-out scoring.