Abstention

The centrepiece. A system that answers everything is wrong at whatever rate the questions happen to produce, and you find out from a user. Cairn chooses a threshold so that the rate of wrong answers among the questions it does answer is bounded by a number you pick, and reports what that cost in coverage.

The guarantee, stated precisely

After cairn calibrate at alpha = 0.1:

With probability at least 0.95 over the calibration draw, the error rate among answered questions on exchangeable new data is at most 0.10.

Every part of that sentence is doing work.

• Among answered questions. Abstentions are not errors. The bound is conditional on the system having committed to an answer.
• On exchangeable new data. The guarantee transfers to questions drawn like the calibration questions. Point the system at a different kind of question and it says nothing. This is the assumption that fails first in practice, which is why the service watches for drift.
• With probability at least 0.95. The bound is over the calibration draw, not a certainty. delta is 0.05.
• At most 0.10. Your choice, in the configuration as calibration.alpha.

How the threshold is chosen

Calibration runs its slice of the golden set with no threshold in force, because the decision being fitted is exactly the one a threshold would pre-empt. Each answered case contributes a pair: the confidence and whether the judge called it correct. Then:

1. Sort by confidence, highest first.
2. For each prefix (the set that would be answered at a threshold of that confidence) compute the empirical error and its one-sided Clopper-Pearson upper bound at delta = 0.05.
3. Take the lowest confidence whose bound is still at or below alpha: the largest answered set that satisfies the bound.
4. If no prefix qualifies, the threshold is above the maximum confidence, the system abstains on everything, and the record says so.

That last case is not a bug, and the harness will produce it on a small calibration set. Five correct answers put the 95 percent upper bound near 0.45, so no threshold can honestly certify 0.10. Refusing to answer is the truthful response to insufficient evidence; the alternative is a promise the data does not support. The fix is more calibration data, not a lower bar.

The confidence signal

The threshold is applied to one scalar built from four signals, weighted and renormalised when a signal is absent:

Signal	Weight	What it is
retrieval	0.4	the top-1 fused score against the candidate list's range, and the top-1 to top-2 margin
rerank	0.3	the reranker's top logit margin, absent when no reranker ran
self	0.2	the model's own reported confidence
citation	0.1	the share of claims whose citations verified

The components are kept on every answer, so a decision can be attributed to the signal that drove it rather than guessed at.

What abstention returns

Not a dead end. The status is abstained, the text is a fixed sentence, the claims are empty, and the passages are still attached. The reader gets the closest material and the honest statement that the system will not commit to a reading of it. That is more useful than a confident wrong answer and more useful than silence.

Reading the result

Four numbers, and they have to be read together:

Number	Reading
error_rate_answered	the rate the guarantee is about
answer_rate	what the guarantee cost; a system that abstains on everything meets any alpha and is useless
unanswerable_abstained_rate	the questions that should be refused, and were
answerable_answered_rate	the questions that should be answered, and were

guarantee_met reports whether the error rate came in at or below alpha on the evaluation slice: the held-out check on the threshold fitted elsewhere. When it is false, the gate refuses the candidate by default.

Choosing alpha

alpha is a product decision, not a tuning parameter. It is the rate of wrong answers you are willing to ship, and it trades directly against coverage: a smaller alpha answers fewer questions. Set it from the cost of being wrong in your setting, calibrate, then read the answer rate and decide whether the trade is acceptable. Both numbers are in the report because neither means anything alone.

Where it breaks

• Exchangeability. New questions must look like the calibration questions. Drift monitoring exists because this assumption decays quietly.
• The judge. The bound is over what the judge called correct. A judge with a systematic blind spot produces a guarantee about that blind spot.
• Calibration size. Small sets give wide bounds, and wide bounds mean abstaining more. Visible in n_calibration and bound_at_threshold on the record.
• Configuration changes. A record fitted for one configuration hash is ignored, with a warning, when the configuration changes. Applying it would promise a guarantee that was never measured for the components now in use.