Write a configuration

A configuration is one YAML file naming a component for each stage. Two files that differ in one line are one comparison, which is the whole point.

name: default
description: Gradient boosting on causal window features, trained on site A.

data:
  raw: ../data/raw
  prepared: ../data/prepared

features:
  name: causal
  params: {short_window: 6, long_window: 24}

model:
  name: lgbm
  params: {n_estimators: 600, learning_rate: 0.05, num_leaves: 63, seed: 7}

split: {seed: 7, train: 0.6, calibration: 0.2, test: 0.2}
train_sites: [A]
test_sites: [A, B]

alert: {target_precision: 0.30, max_alerts_per_patient_day: 4.0, confidence: 0.95}

paths: {reports: ../reports, artifacts: ../.prodrome}

Paths resolve against the configuration file's own directory, so a configuration works the same wherever it is invoked from.

Quote a description containing a colon

description: Baseline: qSOFA is invalid YAML. A shipped configuration once had exactly that, prodrome bench skipped it with a warning on stderr, and the leaderboard silently omitted four of five models. Every shipped configuration is now loaded by the test suite for that reason.

The shipped set

File	Model	What it is for
default.yaml	LightGBM on 205 causal features	the model the gate protects
clock.yaml	ICU hour alone	the trivial baseline every row is read against
qsofa.yaml	partial qSOFA	the bar a ward already clears
sirs.yaml	the four SIRS criteria	the same, older and less specific
logistic.yaml	regularised logistic regression	is the boosting earning its complexity?

prodrome doctor lists everything registered:

feature_set  causal
model        clock, gru, lgbm, logistic, qsofa, sirs, sofa

The fields that matter

train_sites and test_sites. A site in test_sites but not train_sites is an external site, and its report is the one that counts. Training patients are split three ways; an external site is never split, because every one of its patients is untouched and using all of them buys the statistical power that makes the number worth quoting.

split. By patient, stratified on outcome, deterministic in the seed. The calibration part exists because a threshold chosen on training data promises nothing.

alert. See Calibrated abstention. Asking for a precision floor a calibration slice cannot certify gets you a policy that alerts on nothing, which is a real answer rather than a failure.

utility. The published scoring constants. They are exposed so they can be read, not tuned. Changing one produces a score comparable with no published result, and the report records that they were changed.

Comparing two of them

prodrome bench --configs 'configs/*.yaml' --out reports/leaderboard.md

Every configuration is trained, calibrated and evaluated on every one of its test sites, and the leaderboard prints external rows first. A configuration whose extra is missing is skipped and listed, not failed.

Adding a component

Register a class under a name and the configuration can select it:

from prodrome.core.registry import register

@register("model", "recency")
class RecencyModel:
    @property
    def name(self) -> str: ...
    @property
    def params(self) -> dict: ...
    @property
    def fitted(self) -> bool: ...
    def fit(self, train, valid=None) -> None: ...
    def predict_risk(self, features): ...
    def save(self, directory) -> None: ...
    def load_from(self, directory) -> None: ...

Two rules. A FeatureSet is stateless and causal per stay; anything needing a population statistic belongs to the model and is fitted on the training split alone. And save must write everything needed to reload, including preprocessing, because a clinical model without its feature pipeline is not a model, it is a liability.