What the model reads¶
A sepsis model can score well for three quite different reasons, and only one of them is the reason anybody wants.
Physiology. The patient's vital signs and laboratory values are drifting the way a septic patient's do. This is the signal a bedside monitor is supposed to provide.
The workup. Somebody ordered a lactate, a blood culture and a repeat white cell count within an hour. Nobody orders those for a patient they are relaxed about. The pattern of which tests exist therefore encodes clinician suspicion, and the label is a record of clinician suspicion, so a model can score by learning the ordering behaviour. That is a real and useful signal in a deployed system, but it is not an early warning: by the time the workup starts, somebody is already worried.
The context. How long the patient has been in the unit, when they were admitted, how old they are. Septic stays in this corpus average 59 hours against 38 for the rest, so "this patient has been here a while" carries information that has nothing to do with their physiology at this hour.
Telling them apart¶
prodrome ablation refits the same model on restricted feature sets and prints the
comparison. On site B, which the model never trained on:
| ablation | features | utility | share of full | auroc | auprc | in window |
|---|---|---|---|---|---|---|
| full | 205 | 0.2468 | 100% | 0.8016 | 0.0724 | 0.153 |
| physiology_only | 125 | 0.1159 | 47% | 0.7697 | 0.0604 | 0.117 |
| process_only | 75 | 0.2159 | 87% | 0.7263 | 0.0523 | 0.109 |
| clock_only | 3 | 0.2428 | 98% | 0.6656 | 0.0488 | 0.074 |
| workup_only | 72 | 0.2119 | 86% | 0.7539 | 0.0520 | 0.115 |
| no_time_index | 202 | 0.2316 | 94% | 0.7937 | 0.0707 | 0.156 |
The process and clock columns alone reach 87% of the full model's utility, more than physiology alone at 47%. This model is substantially reading the workup rather than the patient, which is worth knowing before anyone calls it an early warning.
The clock is the uncomfortable row¶
Three features, nothing learned, 98 percent of the utility score. And it is not an artefact of the model: alerting on every patient from ICU hour 48, with no model at all, scores +0.227 on site B.
That is a statement about the metric as much as about the models. The challenge utility score is a weak discriminator between a real model and a trivial one, on external data, if you read it alone.
What separates them is everything else in the table. Against the clock, the trained model has:
| clock | default | ratio | |
|---|---|---|---|
| AUROC | 0.628 | 0.802 | ranking is far better |
| detection inside the window | 0.065 | 0.153 | 2.4 times as many |
| calibration error | 0.097 | 0.003 | 30 times smaller |
A model whose probabilities mean something can support a calibrated threshold. One whose score is a rank cannot. That is the difference the headline number hides, and it is why the leaderboard is deliberately wide.
Why this is reported rather than fixed¶
The obvious move is to drop the clock and the staleness columns and publish the
physiology-only model. no_time_index shows what that costs: utility 0.2316 rather
than 0.2468, and in fact a slightly better in-window detection rate.
But removing them would not make the finding go away, it would hide it. The workup signal is real, it is available at the bedside, and a deployed system would be foolish to ignore it. What matters is that nobody calls the result an early warning without knowing where it came from, which is why the ablation is a first-class command and its verdict is printed on the model card.