Golden set reference¶

The golden set is the frozen half of every experiment. It is what makes two runs comparable, and its version is recorded on every report so two reports built on different sets are never compared.

Format¶

JSONL, one case per line, with a golden.version file beside it holding a version string.

{"id": "ret-014", "question": "How many days does a customer have to request a credit for a damaged delivery?", "answerable": true, "expected_answer": "14 days", "support": [{"doc": "returns-and-credits-policy.md", "quote": "A customer has 14 days from delivery to request a credit"}], "tags": ["numeric", "policy"], "notes": ""}
{"id": "una-006", "question": "What is the name of the chief executive of Halcyon Provisions?", "answerable": false, "support": [], "tags": ["unanswerable"], "notes": "The corpus never names an executive."}

Field	Type	Required	Meaning
`id`	string	yes	Unique; a duplicate is an error at load
`question`	string	yes	Asked exactly as written
`answerable`	bool	no, default `true`	`false` means the correct behaviour is to abstain
`expected_answer`	string	when answerable	Short and specific: a number, a date, a name, a short phrase
`support`	list	when answerable	Verbatim quotes, each naming the document that contains it
`tags`	list	no	`numeric`, `date`, `name`, `policy`, `multi_doc`, `unanswerable`, `false_premise`
`notes`	string	no	Why this case exists; never used in scoring

support[].quote must appear verbatim in support[].doc. It is how the runner locates the supporting chunk, which is the denominator of every retrieval metric. support[].doc is the document's basename.

The shipped set¶

120 cases over the Halcyon Provisions corpus:

90 answerable, each with a verbatim support quote that exists in exactly the named document, and a short expected answer.
30 unanswerable: 15 about things the corpus genuinely does not contain, and 15 with a false premise, which is the harder failure because the question presupposes something the documents contradict.

The unanswerable third is not padding. Without it, abstention cannot be measured at all, and a system that answers everything would look identical to one that knows when to stop.

Why expected answers are short¶

Every answer metric is a comparison against this string. A paragraph-long reference makes token F1 measure verbosity rather than correctness, and makes exact match meaningless. A number, a date, a name or a short phrase keeps the metric about the fact, and lets a system that says it in different words still score.

Writing your own¶

The single most useful thing you can do with Cairn is point it at your documents and write a golden set for them. Numbers on the sample corpus are numbers about the sample corpus.

Sample real questions. Take them from what people actually ask, not from what the documents happen to say. A set written by reading the corpus tests retrieval of the passages you were looking at.
Write the answer first, then find the quote. If you cannot find a verbatim supporting quote, the case is not answerable from the corpus, and that is a finding.
Include a quarter unanswerable cases, with false premises among them. Questions whose premise the documents contradict are where confident wrong answers come from.
Keep answers short and specific.
Aim for at least 100 cases. Conformal calibration needs a slice of them, and small calibration sets give wide bounds, which means abstaining more. With 40 percent held out, 100 cases is roughly 40 to fit a threshold on.
Freeze it and version it. Bump golden.version on any change. The gate refuses to compare reports across versions, which is the protection against quietly editing the exam.

Never ship a golden set built from private documents

A golden set built from a private corpus contains verbatim quotes from it. It is a leak wearing a test's clothes. Keep it wherever the documents live, and publish numbers rather than the set.

The split¶

cairn calibrate splits the set, stratified on answerable and deterministic in calibration.seed:

the calibration slice fits the abstention threshold;
the eval slice measures it.

The stratification matters: a calibration slice that happened to contain few unanswerable cases would fit a threshold that has never seen the questions it most needs to refuse. Because the split is seeded, the two slices are the same on every machine, so a threshold and the report that checks it are reproducible.

cairn eval --split all scores everything including the calibration rows. It is useful for a quick look and is never a promotion decision, because the threshold was fitted on part of what it is being scored against.

Regenerating the sample¶

cairn corpus generate --out data/sample --seed 7

Deterministic from the seed. The generator is committed and its output is not: nothing under data/ is stored in the repository or the package. The command checks the corpus it wrote against a pinned digest, so a change to the generator shows up as a one-line change to that digest, and reports made on different corpora can never be compared by accident because the gate refuses them.