Skip to content

Evaluate and gate

The workflow that turns a change into evidence: produce a report, compare it against the one you are shipping today, and let the exit code decide.

The cycle

cairn ingest    --config configs/default.yaml
cairn calibrate --config configs/default.yaml
cairn eval      --config configs/default.yaml

Each step depends on the previous one and refuses to guess:

  • ingest writes the index and its manifest.
  • calibrate fits the abstention threshold on the calibration slice and writes the record, stamped with the configuration hash.
  • eval runs the evaluation slice at that threshold and writes reports/<config>-<timestamp>.json plus reports/latest/<config>.json.

The split is stratified on answerability and deterministic in calibration.seed, so the evaluation slice never contains a row the threshold was fitted on.

Comparing two runs

cairn gate --baseline reports/latest/default.json --candidate reports/ci/default.json

Exit 0 promotes, exit 1 rolls back. Every reason is a sentence with the numbers in it, so a failing job explains itself without anyone opening the JSON:

roll back: default against default on judge_score
- judge_score dropped by 0.329 (baseline 0.902, candidate 0.573); the allowed drop is 0.000.
- the candidate's abstention guarantee is not met: error rate among answered is 0.520 against alpha 0.350.

What the gate checks

Check Default Flag
primary quality dropped no drop allowed --max-quality-drop
citation validity dropped no drop allowed --max-citation-drop
cost per question rose 15 percent --max-cost-increase
p95 latency rose 25 percent, and at least 50 ms --max-p95-increase, --min-p95-delta-ms
candidate met its abstention guarantee required --no-require-guarantee
same corpus and golden version always required none

The primary quality metric is the judge score when both reports have one, and token F1 otherwise, because comparing a rubric grade against a lexical overlap would be meaningless.

The last row cannot be waived. If the corpus hash or the golden set version differ, the two reports did not measure the same thing and no delta between them means anything, so the gate fails and says so.

Cost and latency rises are relative. When the baseline is zero, as it is with the stub's cost, any positive candidate value is an unbounded rise and is reported that way rather than divided by zero.

Latency has two more rules, because a timing is a property of the machine as much as of the code. A rise smaller than --min-p95-delta-ms never blocks, whatever the percentage: the stub answers in single-digit milliseconds, and 2 ms to 4 ms is a 100 percent rise that means nothing. And every report records the machine it was measured on (notes.environment: system, architecture, Python, CPU count). When the baseline and the candidate come from different kinds of machine, latency is reported but not gated, and the gate says so in a note. That is exactly the CI case: the committed baseline was measured on a laptop and the candidate runs on a shared runner, so their timings are not comparable and are not compared.

Prove the gate works

A gate nobody has seen fail is a gate nobody should trust. Degrade the stub on purpose:

cairn eval --config configs/default.yaml --out reports/good.json
CAIRN_STUB_DEGRADE=0.5 cairn eval --config configs/default.yaml --out reports/degraded.json
cairn gate --baseline reports/good.json --candidate reports/degraded.json; echo "exit $?"

The last line prints exit 1.

In CI

The shipped workflow keeps the committed reports/latest/default.json as the baseline, runs the pipeline on the commit, and gates the new report against it:

- name: Keep the committed baseline before this run overwrites it
  run: cp reports/latest/default.json /tmp/baseline/default.json

- run: cairn corpus generate
- run: cairn ingest    --config configs/default.yaml
- run: cairn calibrate --config configs/default.yaml
- run: cairn eval      --config configs/default.yaml --out reports/ci/default.json
- run: cairn attack    --config configs/default.yaml --reports-dir reports/ci
- run: cairn gate --baseline /tmp/baseline/default.json --candidate reports/ci/default.json

Two things make this cheap enough to run on every commit: the default configuration uses the deterministic stub, so there is no key and no spend, and reports are committed, so the history of quality is visible in the repository rather than in someone's terminal. The corpus itself is generated at the top of the job and checked against its pinned digest, so the baseline and the candidate are measured on identical text or the gate refuses to compare them.

Comparing configurations

cairn bench --configs 'configs/*.yaml' --out reports/leaderboard.md

bench ingests, calibrates and evaluates each configuration in turn and writes the leaderboard sorted by the primary metric. A configuration that cannot run (an optional extra missing, a provider unreachable) is skipped and listed at the end rather than stopping the run; progress goes to stderr so --json output stays parseable.

Use bench to choose, and gate to defend the choice afterwards.

Practical notes

  • Re-calibrate when the configuration changes. A record fitted under a different configuration hash is ignored with a warning, and the run continues uncalibrated. notes.threshold_source in the report says which happened.
  • --split all scores the whole golden set, including the calibration rows. Useful for a quick look, never for a promotion decision.
  • Judge scores are only comparable within one rubric version, which is recorded in the report's notes.
  • Keep the golden set frozen. Changing it changes its version, and reports either side of the change stop being comparable, which the gate enforces.