Calibrated abstention¶
A risk score is not a decision. The decision is whether to raise an alarm, and its two costs are not symmetric: a missed sepsis can kill someone, and a false alarm makes the next alarm slightly less likely to be believed.
The problem with a default threshold¶
Alarm fatigue is the documented reason early-warning systems fail on a ward rather than in a paper. A model that ships with 0.5, or with whatever maximised F1 on a validation set, has not made the trade, it has avoided it, and the operator inherits a system whose alarm burden nobody has bounded.
What Prodrome chooses¶
The operator states two numbers in the configuration:
alert:
target_precision: 0.30 # of alerted patients, how many must be right
max_alerts_per_patient_day: 4.0 # how many alarms a ward will absorb
confidence: 0.95
On a calibration slice of patients the model has never seen, the threshold sweep evaluates every candidate cut and takes the lowest one where both hold:
- the precision among alerted patients, at its exact Clopper-Pearson lower bound, clears the floor;
- the alert rate stays inside the budget.
Lowest, because among the thresholds that keep the promise, the most sensitive one catches the most patients.
The bound is Clopper-Pearson rather than a normal approximation because a calibration slice of a few thousand patients yielding a few hundred alerts is not large enough for an asymptotic interval to be honest.
What it promises¶
The record states it in one sentence a clinician can read:
With probability at least 95% over the calibration draw, at most 4.0 alerts are raised per patient-day and at least 30% of alerted patients go on to develop sepsis.
Why 30 percent and not 90: about 7 percent of these patients ever become septic, so a floor of 0.30 is close to a fourfold enrichment over the base rate, and it is a floor a calibration slice of this size can actually certify. Asking for 0.90 and receiving a policy that alerts on nothing teaches nobody anything.
What happens when it cannot be kept¶
attainable is set to false, the threshold is placed above every reachable risk, and
the system alerts on nothing.
This matters more than it sounds. The tempting alternative is to return the best available threshold with a note, after which every downstream consumer treats a guarantee that was never certified as if it were true. A guarantee that cannot be certified must not be implied.
On the real corpus this is not hypothetical. Neither qSOFA nor SIRS can certify 30 percent precision at any threshold inside the budget, so both alert on nothing and score exactly zero utility. That is the honest output.
Whether the promise survives¶
Calibration happens at the training hospital, which is the realistic story: a hospital calibrates on its own data before switching the thing on. The evaluation then checks the promise on patients from a hospital that was never involved, and the report says what happened either way.
| model | precision on A (calibrated here) | precision on B (never seen) | held |
|---|---|---|---|
| clock | 0.506 | 0.346 | yes |
| default | 0.332 | 0.305 | yes, barely |
| logistic | 0.311 | 0.166 | no |
The logistic model kept its promise at home and broke it elsewhere. Nothing in its internal report predicted that, which is the argument for measuring it rather than assuming it.
An alert is an event, not an hour¶
If the risk sits above the threshold for fourteen hours, a ward sees one alarm and then a patient who is already flagged. Counting hours would make every model look unusable and would rate a model that flickers around the threshold identically to one that latches.
So an alert is a rising edge. That has a consequence worth knowing: the alert rate is not monotone in the threshold, because lowering it can merge two separate episodes into one continuous one and reduce the count. The set of thresholds satisfying a budget is therefore not an interval, which is why the sweep evaluates every candidate rather than binary searching.