Quickstart

Seven commands, no API key, no network. The sample corpus belongs to a fictional food distributor, Halcyon Provisions. It is not stored anywhere: cairn init writes it from code together with a 120-question golden set and checks the result against a pinned digest, so the copy on your machine is exactly the copy the published numbers were measured on.

Install

pip install "cairn-pipeline[dev]"

The dev extra brings the test tools, the service and the PDF renderer the sample corpus needs. The base package is enough to ingest, answer and evaluate with the stub provider once a corpus exists. See Install for the other extras.

1. Initialise a directory

cairn init

This writes every shipped configuration to configs/ and generates data/sample/: 30 documents across Markdown, HTML, PDF, DOCX and plain text, the golden set, and a poisoned copy of the corpus for the adversarial suite. The generator refuses to finish unless the corpus hashes to the pinned value, so a drift in a parser or in the generator is an error here rather than a number nobody can reproduce.

Every command below defaults to configs/default.yaml, so --config is left out; pass it to try another configuration.

2. Build an index

cairn ingest

index for default
  documents     30
  chunks        144
  corpus hash   acb06e55ed87073ae65503a9484e4c5bbc470df3a9cead9c2e63fcecdefdbae5
  chunker       sentence_window {'window': 6, 'overlap': 2}
  embedder      hashing {'dim': 512}
  dimensions    512
  backend       numpy
  directory     .cairn/index/default

The index directory carries a manifest: the corpus hash, the chunker and its parameters, the embedder and its dimensions. A query-time configuration that does not match it refuses to load rather than quietly returning nonsense from the wrong vector space.

3. Ask something

cairn ask "What was the average fuel use across the fleet in March 2026?"

Average fuel use across the fleet in March 2026 was 9.4 litres per 100 kilometres.

1. Average fuel use across the fleet in March 2026 was 9.4 litres per 100 kilometres. (verified)
   meeting-minutes-2026-04-15.txt page 1: 'Average fuel use across the fleet in March 2026 was 9.4 litres per 100 kilometres.'

status answered, confidence 0.720, uncalibrated, 219 ms, 0.00000 USD

The answer comes back with a claim, and the claim carries the span it rests on: the source file, the page, and the quote itself. The quote is checked against the chunk it cites before you see it, which is what verified means. The last line says uncalibrated because no threshold is in force yet; the next step fixes that.

4. Calibrate the abstention threshold

cairn calibrate

calibration for default
  alpha                  0.350
  threshold              0.5230
  calibration cases      47
  answered at threshold  13
  empirical error        0.0769
  upper bound            0.3163
  method                 conformal_risk_control

With probability at least 0.95 over the calibration draw, the error rate among
answered questions on exchangeable new data is at most 0.35.

This runs the calibration slice of the golden set with no threshold in force, grades what came back, and chooses the lowest confidence whose finite-sample error bound still sits at or below alpha. From here on cairn ask applies the threshold: the fuel question above still answers at 0.720, and a question whose confidence falls below 0.523 gets a refusal with the closest passages attached.

The target is 0.35 rather than 0.10 for a reason worth reading once: with 47 calibration questions, the tightest 95 percent upper bound on zero observed errors is about 0.31, so a smaller target cannot be certified at this sample size, and the procedure refuses to pretend otherwise. When no threshold can satisfy the target at all, the record says so and the pipeline abstains on every question. See Abstention.

5. Produce a report

cairn eval

The report lands in reports/ as a timestamped file and in reports/latest/default.json, which is the file the gate compares against. It holds retrieval metrics (recall, nDCG, MRR), answer metrics (exact match, token F1, judge score, citation validity, faithfulness), abstention metrics (answer rate, error rate among answered, whether the guarantee held), and operational metrics (cost per question, p50 and p95 latency, tokens, cache hit rate).

6. Try to make it worse

The point of the harness is that a regression cannot be argued with. Degrade the stub on purpose, evaluate again, and put the two reports in front of the gate:

cairn eval --out reports/good.json
CAIRN_STUB_DEGRADE=0.5 cairn eval --out reports/degraded.json
cairn gate --baseline reports/good.json --candidate reports/degraded.json

The gate exits 1 and says why, with the numbers in the sentence:

roll back: default against default on judge_score
- judge_score dropped by 0.329 (baseline 0.902, candidate 0.573); the allowed drop is 0.000.
- the candidate's abstention guarantee is not met: error rate among answered is 0.520 against alpha 0.350.

That exit code is the whole idea. Wire it into CI and a change that makes the system worse cannot be merged by accident. See Evaluate and gate.

7. Run the adversarial suite

cairn attack

Fourteen questions are asked against a copy of the corpus carrying six poisoned documents, each with an injected instruction and a canary string. The report says whether each document was retrieved, whether its canary or the system prompt leaked into an answer, and what the scrub removed. The shipped run retrieved all six and leaked nothing; the command exits 1 the moment that changes. See Injection.

Next

• Serve it: an HTTP endpoint with Prometheus metrics and a span per stage.
• Write a config: the file that defines an experiment.
• Add a component: a new chunker, embedder, reranker or provider behind the existing interfaces.