Write a config

One YAML file describes one complete pipeline, and the configuration is the unit of comparison: the leaderboard is a row per configuration, the gate compares two, and an index manifest records which one built it. Anything you might vary between runs lives in this file, and nothing else does.

A complete file

name: hybrid-local
description: Hybrid retrieval with a local embedder and cross-encoder reranking.
corpus: ../data/sample/corpus
golden: ../data/sample/golden.jsonl

ingest:
  chunker:
    name: sentence_window
    params: {window: 6, overlap: 2}
  include: ["**/*.md", "**/*.txt", "**/*.html", "**/*.pdf", "**/*.docx"]
  exclude: []

embedding:
  name: sentence_transformers
  params: {model: all-MiniLM-L6-v2}

index:
  backend: numpy
  dir: ../.cairn/index

retrieval:
  mode: hybrid
  k: 8
  candidates: 40
  rrf_k: 60
  rewrite: {enabled: false, n: 3}
  rerank:
    name: cross_encoder
    params: {model: cross-encoder/ms-marco-MiniLM-L-6-v2}
    enabled: true
    top: 40

generation:
  provider: {name: stub}
  max_tokens: 600
  temperature: 0.0

calibration: {alpha: 0.10, split: 0.4, seed: 7, dir: ../.cairn/calibration}
judge: {name: stub}
confidence:
  name: weighted
  params: {w_retrieval: 0.4, w_rerank: 0.3, w_self: 0.2, w_citation: 0.1}
budget: {max_cost_per_question_usd: 0.02, max_p95_latency_ms: 8000}

Relative paths resolve against the directory the file lives in, so a configuration in configs/ refers to the corpus as ../data/sample/corpus and the whole repository moves without editing anything.

The fields that change results

name

Lowercase, digits, dots, dashes and underscores. It names the index directory, the calibration record and the report file, so two configurations never overwrite each other's artefacts.

ingest.chunker

Chunker	Parameters	Use when
fixed	size, overlap in characters	a baseline, or text with no structure
sentence_window	window, overlap in sentences	the default; keeps sentences whole
structure	none	headings and tables matter, and the format has them

Chunking is the single change that moves retrieval most, and the honest way to choose is to run all three through the harness rather than to reason about it.

retrieval

mode is dense, bm25 or hybrid. Hybrid pulls candidates from each index and fuses them by reciprocal rank with the constant rrf_k, then cuts to k, which is how many passages generation sees. Larger k costs input tokens on every question, which the report prices.

rewrite.enabled asks the provider for n paraphrases and fuses every result list. It is the biggest recall win on vague questions and it costs a model call per question: exactly the kind of trade the report exists to settle.

rerank re-scores the candidates with a model that sees the question and the passage together. top is how many it looks at, defaulting to candidates.

generation.provider

stub, ollama, openai or anthropic. abstain_text is the sentence an abstention returns.

calibration

alpha is the target error rate among answered questions, split is the share of the golden set held out to fit the threshold, and seed fixes the stratified split. See Abstention.

judge

stub grades mechanically and free; llm applies the versioned rubric through a provider. Give it its own provider when you want a different model from the one being graded:

judge:
  name: llm
  params:
    provider: {name: anthropic}

budget

Advisory, recorded in the report's notes as cost_within_budget and p95_within_budget. The enforcing comparison is the gate's.

What is deliberately not in the file

API keys and hosts. Those come from the environment (OPENAI_API_KEY, ANTHROPIC_API_KEY, OLLAMA_HOST), so a configuration is safe to commit and two machines running the same file run the same experiment.

The configuration hash

content_hash() covers every field that changes behaviour, and deliberately excludes the paths, the description and the directories, so the same experiment on two machines hashes the same. It is stamped on the report and on the calibration record; a record whose hash no longer matches is ignored with a warning rather than applied to a pipeline it was never measured on.

Changing one thing at a time

An ablation is a set of files differing in one field. The four shipped hashing configurations differ only in how candidates are combined:

File	Difference
configs/bm25.yaml	lexical only
configs/dense-hashing.yaml	dense only
configs/hybrid-hashing.yaml	both, fused
configs/default.yaml	the same as hybrid, and what CI runs

cairn bench --configs 'configs/*.yaml' --out reports/leaderboard.md

Check a file before you trust it

cairn doctor --config configs/hybrid-local.yaml

doctor reports whether the file parses, whether the corpus and golden set exist, whether an index has been built, whether a calibration record is present, and whether the components it names are installed.