Cairn¶

A cairn is a stack of stones marking a route across ground that offers nothing to follow. It does two things at once: it points the way, and it proves somebody walked here before you and checked.

Cairn answers questions from your documents, cites the exact span it used, refuses to answer when the evidence is thin, and can prove every one of those behaviours with a number.

Better than what we had¶

A frozen evaluation set, one protocol, and a promotion gate that exits non-zero when a candidate is worse. A change to the chunk size stops being a matter of taste.

Not in the documents¶

The abstention threshold is calibrated by conformal risk control, so the rate at which answers are wrong is bounded by a number you choose. Below the threshold you get the closest passages and a refusal.

Where it came from¶

Every claim carries a span-level citation, and the citation is verified rather than trusted. A claim whose quote does not support it is caught by the harness, not by the reader.

The leaderboard¶

Generated by cairn bench over the sample corpus and its 120-question golden set. Every configuration is the same corpus and the same protocol, so the rows are comparable to each other and to nothing else. The stub provider is deterministic and extractive, so these numbers need no API key and CI reproduces them on every commit.

config	recall@5	nDCG@10	token F1	judge	citation validity	answer rate	error among answered	p95 ms
bm25	0.963	0.895	0.319	0.914	1.000	0.500	0.194	2
default	0.870	0.846	0.334	0.902	1.000	0.347	0.240	2
hybrid-hashing	0.870	0.846	0.334	0.902	1.000	0.347	0.240	2
dense-hashing	0.815	0.745	0.340	0.898	1.000	0.472	0.206	2

Sorted by judge. Golden set version 1; corpus hash acb06e55ed87. Generated 2026-09-05 09:20 UTC.

default is hybrid-hashing under its shipped name. Two things worth reading off the table: BM25 alone beats the hashing hybrid on recall, because a hashing embedder is not semantic and the golden questions reuse the documents' own vocabulary; and every configuration meets the guarantee it states, error among answered questions at or below 0.35. Why 0.35 and not 0.10 is explained in Abstention and in the comment at the top of configs/default.yaml.

With a real embedder, installed by the local extra (weights download on first use, so CI does not run these):

config	recall@5	nDCG@10	judge	p95 ms
dense-local	0.926	0.873	n/a	10
hybrid-local	0.926	0.903	n/a	266
hybrid-local-norerank	0.926	0.892	n/a	11

Sorted by token_f1. Golden set version 1; corpus hash acb06e55ed87. Generated 2026-09-05 09:28 UTC.

Semantic retrieval lifts dense recall from 0.815 to 0.926, and the cross encoder gives the best ranking of all at roughly a hundred times the latency. The answer columns are zero, and that is the guarantee working rather than failing: with these confidence signals the calibration slice could not certify the 0.35 bound at any threshold, so the record abstains on everything instead of answering at whatever confidence happened to be highest.

Reproduce all of it with no API key and no network:

pip install "cairn-pipeline[dev]"
cairn init
cairn bench --configs 'configs/*.yaml' --out reports/leaderboard.md

cairn init writes the configurations and generates the corpus; nothing under data/ is stored in the repository or the package. Configurations whose extra or key is missing are skipped and listed at the end, not failed.

What it is¶

A pipeline and a small library, not an application. No chat interface, no accounts, no document management. It ingests a corpus, builds a versioned index, answers a question with citations or an abstention, and produces a report you can compare against last week's.

It runs three ways: entirely local with a local model and a local embedder, against a hosted model, or in a no-model mode where a deterministic stub stands in, so the whole pipeline can be tested with no key and no network. That last mode is how the tests and the published numbers run.

documents  ->  parse  ->  chunk  ->  embed  ->  hybrid search  ->  rerank
                  |                                                   |
             provenance                                        injection scrub
             (page, span)                                             |
                  |                                          generate with
             versioned index                                 structured citations
                  |                                                   |
                  |                                            confidence score
                  |                                                   |
                  |                              above threshold -> answer and spans
                  |                              below threshold -> abstain and passages
                  v
          evaluation harness  ->  report  ->  promotion gate

Where to start¶

Quickstart: index the sample corpus, ask a question, produce a report, in about five minutes.
Why measure: what the harness measures and why each number is there.
Evaluate and gate: wiring the gate into CI so a regression cannot be merged.

What it deliberately does not do¶

No chat interface, no user accounts, no document management, no fine-tuning. The value is the measured pipeline; an hour spent on a interface is an hour not spent on the thing that makes this different.