Why measure¶

There are thousands of retrieval pipelines, and almost all of them stop at the demo: ingest, embed, retrieve, generate, screenshot. A demo cannot answer the three questions anyone running this in production actually asks. Cairn is built around answering them with numbers.

Is it better than what we had?¶

Without an evaluation set, a change to the chunk size or the embedding model is a matter of taste. Cairn fixes the comparison:

A frozen golden set. Questions, expected answers and the supporting quote, versioned with the corpus. The report records which version produced it, and the gate refuses to compare two reports built on different ones.
One protocol. The same runner, the same metric definitions and the same judge rubric for every configuration.
A configuration as the unit of comparison. One YAML file describes one complete pipeline. Two runs of the same file over the same corpus are the same experiment, which is what config_hash and corpus_hash in the report are for.
A gate. cairn gate exits non-zero when the candidate is worse. That exit code is the difference between a tool and a dashboard.

What happens when the answer is not in the documents?¶

Most systems answer anyway, fluently and wrongly. Cairn calibrates a threshold so that the error rate among answered questions is bounded, and reports the answer rate alongside it, because a system that abstains on everything is useless and the number has to be visible. See Abstention.

Can you show me where that came from?¶

Most systems paste a list of source documents next to the answer and hope. Cairn attaches span-level citations and then verifies them. See Citations.

What the report contains¶

Four groups, each answering a different question about the same run.

Retrieval: did the evidence come back at all?¶

Measured over the cases whose support quote was located in the chunk store, so the denominator is honest even when retrieval missed.

Metric	Definition
`recall@k`	the supporting chunk is in the top `k`, for `k` in 1, 3, 5, 10
`nDCG@10`	with one relevant chunk this reduces to `1 / log2(rank + 1)` inside the cutoff
`MRR`	mean of `1 / rank`, zero when the chunk never surfaced
`n_with_support`	how many cases had a locatable support quote, the denominator above

Answer: was it right, and was it grounded?¶

Measured over answered, answerable cases.

Metric	Definition
`exact_match`	equal after the standard normalisation: lowercase, drop punctuation, drop articles, collapse whitespace
`token_f1`	the same normalisation, then multiset token overlap; the definition readers will compare against
`judge_score`	mean of a versioned rubric's 0 to 1 grade, or absent when no judge ran
`citation_validity`	verified claims over all claims
`faithfulness`	share of answers where every claim verified

Abstention: was the refusal honest?¶

Metric	Definition
`answer_rate`	answered over all cases
`error_rate_answered`	judged wrong over answered; an answered unanswerable case is wrong by definition
`guarantee_met`	whether `error_rate_answered` came in at or below `alpha`
`unanswerable_abstained_rate`	share of unanswerable cases that abstained
`answerable_answered_rate`	share of answerable cases that answered

The last two exist so nobody can hit a good error rate by refusing everything: they show the two ways abstention can be wrong, and both are visible.

Operational: what did it cost?¶

Metric	Definition
`cost_usd_total`, `cost_per_question_usd`	summed from each provider's own price table
`latency_p50_ms`, `latency_p95_ms`	over all cases, interpolated percentiles
`tokens_total`	input, output and cache tokens
`cache_hit_rate`	share of calls whose stable prompt prefix matched the previous call's

Cost and latency are in the gate, not just the report. A candidate that is a little better and three times the price does not promote by default. Latency is gated only between reports measured on the same kind of machine, and only above a noise floor, because a timing describes the hardware as much as the code.

The judge is evidence, not an oracle¶

The graded metric comes from a model applying a rubric, so the rubric is versioned in the repository and its version is recorded on every verdict and in every report. When the rubric changes, its version changes, and reports either side of that change are not comparable. The default judge is mechanical, not a model at all, so a run in CI grades the same way every time.

What measurement does not do¶

A number on the sample corpus says the pipeline works on the sample corpus. Point it at your own documents and build your own golden set before you believe anything about yours; the harness is what makes that possible, not a substitute for it. See Golden set.