Evaluate and gate¶

The gate is what makes the harness a tool rather than a dashboard. It compares two reports and exits non-zero when the candidate is worse, so a regression cannot be merged by accident.

The three stages¶

prodrome train      # fits on the training patients of the training site
prodrome calibrate  # fits the alert policy on held-out patients of the same site
prodrome eval       # scores every test site and writes one report each

The split is by patient and never by row, stratified on outcome, and deterministic in split.seed. A stay whose first day trains a model and whose second day tests it is a leak that inflates every number.

Reports land in reports/ timestamped and in reports/latest/<config>-site<X>.json, which is the file CI keeps as the baseline. Note that the site is part of the name: two sites of one configuration must not overwrite each other.

Comparing two runs¶

prodrome gate \
  --baseline  reports/latest/default-siteA.json \
  --candidate reports/ci/default-siteA.json

Exit 0 promotes, exit 1 rolls back. Every reason is a sentence with the numbers in it, so a failing job explains itself without anyone opening the JSON:

roll back: default@A against default@A
- utility dropped by 0.1500 (baseline 0.3338, candidate 0.1838); the allowed drop is 0.0000.
- the alert rate rose by 82.0% (0.17 to 0.31 per patient-day); the allowed rise is 15%.
  A ward pays for this in ignored alarms.

What the gate checks¶

Check	Default	Flag
utility dropped	no drop allowed	`--max-utility-drop`
AUPRC dropped	no drop allowed	`--max-auprc-drop`
median lead time fell	0.5 hours	`--max-lead-time-drop-h`
calibration error rose	0.02	`--max-ece-increase`
alert rate rose	15 percent	`--max-alert-rate-increase`
the candidate kept its stated promise	required	`--no-require-guarantee`
the internal-to-external gap widened	0.02	`--max-external-gap-increase`
same corpus, seed and test site	always required	none

The last row cannot be waived. If the corpus digest, the split seed or the test site differ, the two reports did not measure the same thing and no delta between them means anything.

The generalisation check¶

This is the one specific to this project, and it needs both pairs:

prodrome gate \
  --baseline           reports/latest/default-siteA.json \
  --candidate          reports/ci/default-siteA.json \
  --baseline-external  reports/latest/default-siteB.json \
  --candidate-external reports/ci/default-siteB.json

A model can improve on the site it trained on while getting worse on the site it did not. Every single-site number calls that an improvement; this check calls it what it is:

- the gap between the internal and the external site widened by 0.0400
  (baseline 0.0870, candidate 0.1270); the allowed widening is 0.0200.
  The candidate has learned its own hospital.

Without the external pair the gate says so in a note rather than passing silently:

note: generalisation was not gated: no matching pair of internal and external reports
was supplied, so a candidate that improved at home while getting worse elsewhere would
not be caught here.

Prove the gate works¶

A gate nobody has watched fail is a gate nobody should trust. Degrade a report on purpose and check:

python - <<'PY'
import json, pathlib
p = pathlib.Path("reports/latest/default-siteA.json")
r = json.loads(p.read_text())
r["utility"]["normalized"] -= 0.2
pathlib.Path("reports/degraded.json").write_text(json.dumps(r))
PY
prodrome gate --baseline reports/latest/default-siteA.json --candidate reports/degraded.json
echo "exit $?"   # 1

The shipped CI workflow does exactly this on every commit, on a synthetic corpus, and fails the build if the gate ever promotes the degraded candidate.

In CI¶

The workflow generates its own synthetic corpus, so no patient data is downloaded, cached or committed anywhere in that job:

- run: prodrome train     --config ci/configs/default.yaml
- run: prodrome calibrate --config ci/configs/default.yaml
- run: prodrome eval      --config ci/configs/default.yaml --reports-dir ci/reports
- run: prodrome gate --baseline ci/reports/latest/default-siteA.json \
                     --candidate ci/reports/latest/default-siteA.json

The committed reports under reports/latest/ are the record of the real numbers and are refreshed deliberately, not by CI, because CI has no corpus to measure them on.