Report reference

Every cairn eval writes one of these. It is the unit of evidence: the leaderboard ranks them, and the gate compares two.

Where they go

Path	Purpose
reports/<config>-<UTC timestamp>.json	the run, kept
reports/latest/<config>.json	the current baseline, overwritten each run
reports/leaderboard.md	written by cairn bench
reports/attack-<config>.json and .md	written by cairn attack

Reports are committed on purpose, so the history of quality lives in the repository rather than in someone's terminal.

Structure

{
  "cairn_version": "0.1.0",
  "config_name": "default",
  "config_hash": "e26592bbd90d0924",
  "corpus_hash": "3f9c...",
  "golden_version": "1",
  "split": "eval",
  "created_at": "2026-01-01T00:00:00Z",
  "n_cases": 72,
  "retrieval": {"recall_at_k": {"1": 0.0, "3": 0.0, "5": 0.0, "10": 0.0}, "ndcg_at_10": 0.0, "mrr": 0.0, "n_with_support": 54},
  "answer": {"exact_match": 0.0, "token_f1": 0.0, "judge_score": null, "citation_validity": 0.0, "faithfulness": 0.0},
  "abstention": {"alpha": 0.1, "threshold": 0.0, "answer_rate": 0.0, "error_rate_answered": 0.0, "guarantee_met": true, "unanswerable_abstained_rate": 0.0, "answerable_answered_rate": 0.0},
  "operational": {"cost_usd_total": 0.0, "cost_per_question_usd": 0.0, "latency_p50_ms": 0, "latency_p95_ms": 0, "tokens_total": 0, "cache_hit_rate": 1.0},
  "per_case": [],
  "notes": {}
}

Identity

Field	Why it is there
cairn_version	the code that produced the numbers
config_name, config_hash	the experiment; the hash excludes paths, so two machines agree
corpus_hash	sha256 over the sorted document path and content hashes
golden_version	from golden.version, or a content hash of the set when that file is absent
split	eval, calibration or all
n_cases	how many cases produced these numbers

The gate fails outright when corpus_hash or golden_version differ, because the two reports did not measure the same thing.

retrieval

Over cases whose support quote was located in the chunk store, counted in n_with_support. The runner searches the whole store, not just what came back, so recall has an honest denominator even when retrieval missed entirely.

Field	Definition
recall_at_k	share of cases whose supporting chunk is in the top k, for 1, 3, 5, 10
ndcg_at_10	one relevant chunk, so 1 / log2(rank + 1) inside the cutoff, else 0
mrr	mean of 1 / rank, 0 when the chunk never surfaced
n_with_support	the denominator above

answer

Over answered, answerable cases.

Field	Definition
exact_match	equal after lowercase, punctuation removed, articles dropped, whitespace collapsed
token_f1	the same normalisation, then multiset token overlap: the SQuAD definition
judge_score	mean of the judge's 0 to 1 grade, or null when no judge ran
citation_validity	verified claims over all claims, across answered cases
faithfulness	share of answers where every claim verified

abstention

Field	Definition
alpha	the target from the configuration
threshold	the threshold in force, null when uncalibrated
answer_rate	answered over all cases
error_rate_answered	judged wrong over answered; an answered unanswerable case is wrong by definition
guarantee_met	error_rate_answered <= alpha, or null when nothing was answered
unanswerable_abstained_rate	unanswerable cases that abstained
answerable_answered_rate	answerable cases that answered

Read the first three together. A high answer rate with a low error rate is the result worth having; either alone can be bought at the other's expense.

operational

Field	Definition
cost_usd_total, cost_per_question_usd	from each provider's own price table; zero with the stub
latency_p50_ms, latency_p95_ms	interpolated percentiles over all cases
tokens_total	input, output and cache tokens
cache_hit_rate	share of calls whose stable prefix matched the previous call's; null with fewer than two calls

per_case

One row per case, which is where a summary number gets explained.

Field	Meaning
case_id, answerable	from the golden set
status	answered or abstained
confidence	the scalar the threshold was applied to
correct	the judge's verdict; null when abstained
token_f1, judge_score, citation_validity	per case
retrieved_support_rank	where the supporting chunk landed; null if it never did
latency_ms, cost_usd	per case
answer_text	what was returned

Sort by confidence to see what sits near the threshold; filter to retrieved_support_rank: null to see what retrieval missed entirely.

notes

Context that is not a metric: the judge and its rubric version, the provider and the models seen, the retrieval mode and depth, whether a reranker or rewriting ran, whether the threshold was calibrated, and the budget check.

threshold_source is worth reading first. It says calibrated or uncalibrated (model abstain flag), and an uncalibrated report carries no guarantee whatever guarantee_met says.

environment records the machine the report was measured on: operating system, architecture, Python version and CPU count. The gate compares latency only between reports from the same kind of machine; across machines it reports the delta in a note and does not block on it.

The leaderboard

cairn bench writes a markdown table sorted by the primary metric, which is the judge when every report has one and token F1 otherwise:

config	recall@5	nDCG@10	token F1	judge	citation validity	answer rate	error among answered	cost/q	p95 ms

The footer records the golden set version and the corpus hash, because a leaderboard whose rows came from different corpora is a chart, not a comparison.