CLI reference¶
Every command takes --config (default configs/default.yaml) and, where the
output is structured, --json. Rich tables are the default because a person is
usually reading; --json is for a script, and the two are never mixed on stdout,
so any --json output can be piped without stripping decoration. Progress and
warnings go to stderr.
Exit codes¶
| Code | Meaning |
|---|---|
0 |
the command did what it was asked |
1 |
a check failed: the gate refused a candidate, or the adversarial suite found a leak. The command ran correctly; the answer is no |
2 |
usage or configuration error: a missing file, an unknown component, a malformed argument |
Only gate and attack can exit 1.
Global options¶
| Option | Effect |
|---|---|
--version |
print the version and exit |
--verbose, -v |
log what each stage is doing to stderr |
cairn init¶
Write the bundled configurations and generate the sample corpus.
One command takes a fresh install to a directory every other command can run in:
configs/ receives every shipped configuration, and data/sample/ receives the
generated corpus, golden set and poisoned copy. Existing configuration files are
kept unless --force is passed, so a local edit survives a re-run. The corpus is
checked against a pinned digest as it is written. Needs the pdfgen extra
(included in dev) for the two PDF documents.
cairn ingest¶
Build the index for a configuration and print the manifest.
Parses the corpus, chunks it, embeds the chunks and writes the index directory
atomically: a crash mid-build leaves the previous index intact. --json prints
the manifest plus the index directory.
cairn ask¶
Answer one question with citations, or abstain with the closest passages.
| Option | Effect |
|---|---|
--threshold |
override the calibrated abstention threshold |
--passages |
print every retrieved passage, not only the cited ones |
Uses the calibrated threshold when a record for this configuration exists.
cairn calibrate¶
Fit the abstention threshold on the calibration slice and write the record.
Writes the CalibrationRecord and the drift baseline next to it, and prints the
guarantee in words. Requires an index. See
Abstention.
cairn eval¶
Run the evaluation slice at the calibrated threshold and write the report.
| Option | Default | Effect |
|---|---|---|
--split |
eval |
which slice to score; all includes the calibration rows and is never a promotion decision |
--out |
timestamped | write the report here instead |
--reports-dir |
reports |
where reports are written |
Always refreshes reports/latest/<config>.json, which is what the gate compares
against.
cairn gate¶
Compare two reports. Exit 0 to promote, 1 to roll back.
| Option | Default |
|---|---|
--max-quality-drop |
0.0 |
--max-citation-drop |
0.0 |
--max-cost-increase |
0.15 |
--max-p95-increase |
0.25 |
--min-p95-delta-ms |
50 |
--require-guarantee / --no-require-guarantee |
required |
A differing corpus hash or golden set version always fails and cannot be waived.
A p95 rise inside --min-p95-delta-ms never blocks, and when the two reports
were measured on different kinds of machine latency is reported in a note rather
than gated. See Evaluate and gate.
cairn bench¶
Ingest, calibrate and evaluate each configuration, then write the leaderboard.
cairn bench [--configs 'configs/*.yaml'] [--out reports/leaderboard.md] [--reports-dir DIR] [--skip-ingest] [--json]
A configuration that cannot run is skipped and listed at the end rather than
stopping the run. --skip-ingest reuses existing indexes.
cairn attack¶
Run the adversarial suite against the poisoned corpus. Exit 1 if anything
leaked.
Writes reports/attack-<config>.json and a markdown summary. The poisoned
directory defaults to poisoned/ beside the corpus. See
Injection.
cairn corpus generate¶
Generate the sample corpus and its golden set from a seed.
Deterministic: the same seed reproduces the same bytes. The generator is
committed and its output never is; the command refuses to finish unless the
corpus hashes to the pinned value in cairn.sample.generate, so every copy in
the world is the copy the leaderboard was measured on. The seed orders the golden
set and leaves the corpus unchanged. cairn init runs this for you.
cairn serve¶
Run the HTTP service.
Needs the serve extra. See Serve.
cairn doctor¶
Report which optional extras are installed and which providers are usable.
Reports the extras, the registered components, the configuration's state (does the corpus exist, is there an index, is it calibrated) and the tracing setup. Contacts only the local model daemon, on its free listing endpoint; hosted providers are reported by whether their client is installed and a key is present. Nothing here spends money.
Environment¶
| Variable | Effect |
|---|---|
OLLAMA_HOST |
the local daemon, default http://127.0.0.1:11434 |
OPENAI_API_KEY |
credential for the openai provider and embedder |
ANTHROPIC_API_KEY |
credential for the anthropic provider |
CAIRN_PRICING_JSON |
override the provider price tables |
CAIRN_STUB_REFUSE |
make the stub refuse, to exercise the abstention path |
CAIRN_STUB_DEGRADE |
0 to 1; make the stub answer wrongly at that rate, to produce a worse report on purpose |
CAIRN_OTEL_CONSOLE |
1 prints spans to stdout |
OTEL_EXPORTER_OTLP_ENDPOINT |
ship spans over OTLP |