Why measure this way¶
There are hundreds of published sepsis early-warning models. Most report an AUROC on a held-out slice of the hospital they were trained on, and that number answers almost none of the questions a ward would ask. Prodrome is built around four that it does answer.
Was the alert worth raising, given when it arrived?¶
An AUROC treats every correctly ranked hour as equally valuable. A ward does not. An alert twelve hours before onset is worth little because nobody would act on it yet, an alert six hours before is worth the most, an alert after onset is worth progressively less, and an alert on a patient who never becomes septic costs a small amount every hour it stands.
The PhysioNet/CinC 2019 challenge encodes exactly that in a scoring function, and it is the primary metric here. Our implementation is checked against a direct transcription of the official code on two thousand random stays, because "the same numbers" is a claim to be tested rather than asserted.
One property of the metric is worth knowing because it is easy to get backwards: a single isolated alert at the optimal hour scores worse than a single late one. The penalty for going quiet through the hours around onset outweighs the bonus for being early. The metric is scoring a monitor that keeps warning, not a lucky guess.
Does it work anywhere but here?¶
The public corpus is two genuinely different hospital systems, so external validation is possible without leaving the data. Prodrome makes it the primary result rather than a robustness appendix: every table reports the training site and the external site side by side, and the leaderboard prints the external row first so a reader cannot quote the flattering number by accident.
The gate goes further and watches the gap between them. A model can improve at home while getting worse elsewhere, which is overfitting to one hospital, and every single-site number would call that an improvement.
How often will it cry wolf?¶
Alarm fatigue is the documented reason early-warning systems fail on a ward rather than in a paper. A model that ships with "we used a threshold of 0.5" has not made that decision, it has avoided it.
Prodrome makes it explicit: the operator states a precision floor and an alert budget, and the calibration procedure either finds a threshold that provably meets both or refuses. See Calibrated abstention.
Is it beating anything?¶
Two baselines are shipped configurations rather than prose.
The bedside scores. qSOFA and SIRS are what a nurse can compute in their head. A model that cannot outscore them has not earned a place at a bedside, and a paper that never shows the comparison is hiding it. Both are named partial where this corpus cannot support the published definition: there is no Glasgow Coma Scale here, so qSOFA can never reach 3 and SOFA covers four of six organ systems.
The clock. A baseline that alerts on nothing but how long the patient has been in the unit. On this corpus it reaches 98 percent of the trained model's utility score, which says more about the metric than about either model, and a leaderboard without that row would let a reader believe the score came from physiology.
What the report actually contains¶
| Group | Metrics |
|---|---|
| Utility | the challenge score, plus the observed, best, inaction and worst totals it normalises |
| Discrimination | AUROC and AUPRC at the hour level, AUROC at the patient level |
| Timing | detection rate ever in time, detection rate inside the scoring window, lead-time quartiles |
| Calibration | Brier score, expected calibration error, the reliability bins |
| Operational | alerts per patient-day, share of patients alerted, precision among alerted patients |
| Guarantee | the promise in words, and whether it held on these patients |
| Subgroups | all of the above by age band, sex and unit type |
Plus the provenance that makes a comparison legitimate at all: the corpus digest, the split seed, the sites trained on, the site tested on, and the machine it ran on.