Drift-watch: ML Model Monitoring Service

The question you cannot answer today

On a fraud system I worked on, I tracked two failure modes separately. The data flowing in could shift, which showed up immediately in a population stability index. Or the model's actual accuracy could decay, which only became visible weeks later, once fraud labels matured through chargebacks and investigations. Collapsing those into one number means you either panic at harmless noise or sleep through a real decline.

That separation worked. But it lived inside one model's pipeline, as a habit rather than a design.

Drift-watch is what happens when you make it the foundation instead. A service that stores predictions from any model, compares them against a registered baseline, computes performance whenever the truth eventually arrives, and knows nothing about the domain it is watching.

The statistics are not the interesting part. Population stability index, Kolmogorov-Smirnov, chi-square, Jensen-Shannon divergence. All decades old, all a few lines of scipy. What I did not expect is how many ordinary decisions stop being ordinary once delayed labels are a first-class citizen rather than an inconvenience.

Two clocks, and a schema that had to admit it

A window of predictions has two lifecycles, not one.

Drift is computed from predictions. Once a window has closed and a short watermark has elapsed, every prediction that belongs in it has arrived. There is nothing left to learn, so drift is computed once and never revised.

Performance is computed from labels. Labels arrive on their own schedule, sometimes months later. So a window's performance stays open indefinitely, recomputed each time new labels land.

I expected this to be a scheduling detail. It turned out to be a schema decision, and the schema is where it shows most clearly. DriftResult carries a functional unique index across window, feature, test method and segment, so a re-run can never duplicate a row. PerformanceResult deliberately carries no such constraint at all. It is designed to hold many rows per window, because every revision after the first is a legitimate correction rather than a duplicate.

Two tables in the same database, one enforcing uniqueness and one forbidding itself from doing so. That asymmetry is the whole design, expressed in about six lines of migration.

It also breaks a question I had assumed was simple. "What did this window score" is not answerable on its own. You also have to know which labels had arrived when you asked. So every evaluation window stores the baseline version it was compared against, a hash of the resolved config in effect at the time, and the label watermark it was computed from. Without those three, two people looking at the same window on different days see different numbers and have no way to explain the gap.

The asymmetry I had to argue myself into

If labels arriving weeks late is the premise of the system, what about predictions arriving late?

My first instinct was symmetry. Late data is late data, so re-evaluate the window and let the drift number improve.

I decided against it, and the reasoning is the part worth keeping. A label arriving weeks after its prediction is not a fault. It is the domain. A prediction arriving late relative to its own event time is something else entirely: a pipeline hiccup. Treating both as normal would mean a published drift chart could silently move under a reader who had already acted on it.

So drift for a window is computed once, at a configured watermark after the window closes, and a straggler prediction is permanently excluded from it. It is still stored, because predictions are never rejected. It increments that window's late arrival count and logs a warning.

That count matters more than the exclusion. A climbing late arrival rate is a data pipeline problem, and a monitoring service that quietly discards the evidence of one is failing at exactly the job it exists to do.

A p-value stops working precisely when you have enough data to trust it

Here is the trap that catches most homegrown monitoring, and I walked into it.

The obvious way to decide whether a feature has drifted is a significance test. Run Kolmogorov-Smirnov, compare the p-value against 0.05, alert if it clears. That is what I built first, and the config field was called ks_alpha.

It does not work at production scale. A p-value answers a narrow question: is there enough data to be confident these two samples are not identical. At the sample sizes a real evaluation window accumulates, the answer is essentially always yes, no matter how small the shift. Significance saturates exactly when you finally have enough data for it to matter. Every window alerts, on every feature, forever.

So thresholds gate on effect size instead. The Kolmogorov-Smirnov D-statistic, bounded in zero to one. Cramér's V for categoricals, which is the chi-square effect size and, unlike the raw statistic, independent of both sample size and category count. Population stability index and Jensen-Shannon divergence never had the problem, since both are already pure effect-size measures with no p-value at all. The p-values are still computed and stored for context. Nothing is ever gated on them.

The rename from ks_alpha to ks_statistic_threshold was not cosmetic, and it is the kind of change that is dangerous because it looks harmless. A 0.05 significance level and a 0.05 KS statistic threshold are entirely different objects. Carrying the number across would have left both shipped profiles reading as perfectly sensible while being nonsense. Every threshold had to be re-derived from what it actually gates on.

The same care applies to the multiple-comparison correction, and this one is easy to get wrong quietly. Benjamini-Hochberg runs across the feature-level tests within one model's single evaluation window, and nowhere wider. A model with fifty features evaluated over thirty windows is thirty independent fifty-comparison experiments, not one fifteen-hundred-comparison experiment. Pooling across windows would silently redefine what the family of tests means and make the false discovery rate guarantee meaningless. Python's type system cannot express "this list came from exactly one evaluation," so it is documented on the function itself.

Silence is the failure mode

The most dangerous thing a monitoring dashboard can display is nothing.

A metric that cannot be computed is not a metric with no drift. Area under the ROC curve is undefined on a window where every label received so far is the same class, which for a rare-event model is a normal Tuesday. A window that was never evaluated at all, because the scheduler was down or a backfill skipped it, is not a window that was evaluated and found quiet. All three of those, handled naively, render as an absence on a chart. An absence reads as calm.

So nothing is ever allowed to just disappear. Every drift and performance result carries a status and, when it is not computable, a reason. Every chart joins against the grid of windows that should exist rather than plotting whatever rows happen to be there. Four states, always visually distinct: computed, not computable, not configured, and missing.

Missing is deliberately the loudest of them. Without it, a genuinely unevaluated window contributes no rows, and the line connecting its neighbours draws straight across the gap with no break and no marker, indistinguishable from uninterrupted quiet monitoring.

That principle propagated further than I expected. A feature that has been uncomputable for several consecutive windows raises its own alert, with its own persistence gate, entirely separate from drift. Not because a broken feature has drifted, but because silence about a broken feature is exactly the failure this service exists to prevent.

A check that cannot fail is not a check

I hit the same mistake twice, in two unrelated parts of the system, and only recognised it as a pattern the second time.

The alerting engine has no persisted pending counter. Instead it recomputes a signal's streak from scratch each time by rescanning recent history. How far back that rescan looks is a configuration value, and my first version derived it from the persistence counts it had to satisfy: look back as far as the longest gate requires.

That is tautologically correct and therefore useless. A bound defined as "enough" is always exactly enough. It can never catch a misconfiguration, which is the only thing a bound like that exists to catch. So the scan window became an independent field with a validator that requires it to exceed the largest persistence count by a margin, and fails loudly at startup if it does not. A profile whose rescan cannot see far enough back to satisfy its own escalation gate would otherwise leave alerts permanently, silently unable to fire.

The second instance was worse, because it was in the verification layer itself. I wrote a script whose entire job is to assert that each demo scenario numerically produced what it claims. It was passing on an empty database. Every check was of the form "no result violates this condition," and zero results violate every condition. A verifier that cannot fail manufactures confidence rather than establishing it. It now checks explicitly that the scenario's data is actually loaded before it checks anything else.

Both bugs are the same shape. A safety check whose own correctness is defined in terms of the thing it is checking. I now look for that shape specifically, and I have started to think it is one of the more common ways monitoring systems fail: not by measuring wrong, but by measuring something that could never have come out any other way.

The demonstration that disproved itself

The claim I most wanted to show is that a global metric can hide a failure confined to one segment. So I built a scenario for it: inject a shift into one region, weighted at fifteen percent of traffic, and watch the global number stay calm.

My first attempt injected a five sigma shift. It blew straight through the global threshold. The demonstration built to prove that aggregates hide things had produced a failure that aggregates did not hide.

The fix was to dial the shift down to one sigma and verify numerically, not to assume. Here is what the verification prints, from a real run:

$ uv run python -m driftwatch.cli demo-verify segment_isolated
[PASS] no unexpected alerts: none
[PASS] global age PSI stays under the fire threshold throughout: max=0.09425502950483428, fire_threshold=0.25
[PASS] APAC segment breaches, fires, and (once the shift ends) resolves: statuses seen for age/region=APAC: ['escalated', 'resolved']

The global reading tops out at 0.094 against a fire threshold of 0.25. The same feature inside the affected segment reaches 2.30. Anyone reading the global number alone concludes nothing is wrong.

The failed first attempt taught me more than the working version. My instinct had been to make the effect large, and a large enough segment failure is not hidden at all. The entire danger of aggregate monitoring lives in a middle band: shifts big enough to matter locally, small enough to vanish globally. I had to find that band by hand, which is how I know it is real rather than rhetorical.

I set one rule for the demo data and kept it. If a scenario did not show what it claimed, I changed the scenario and regenerated. I never touched the output.

The labels that carried no signal

That rule was tested almost immediately.

The concept drift scenario is meant to show performance degrading while input distributions stay perfectly stable, visible only after labels backfill. I built it, ran it, and got nothing. Not a small effect. Nothing.

The cause was in my own generator. Labels are drawn as a Bernoulli trial on the prediction score, and with a moderate unimodal score distribution that mechanic produces labels with almost no rank correlation to the score at all. Area under the precision-recall curve pins near 0.5 regardless of how much noise gets injected. There was nothing for the degradation event to degrade.

I changed the score distribution to a U-shaped Beta, where scores cluster near zero and one, and the signal appeared:

[PASS] PR-AUC healthy before label_noise, degraded after (final backfilled values):
       early_mean=0.874, late_mean=0.386, clear_threshold=0.65, fire_threshold=0.5
[PASS] initial evaluation frequently lacks enough labels: 57 windows initially not_computable

That last line is the one I care about. Fifty-seven windows had too few labels to evaluate at the moment they closed. The degradation was invisible in real time and only surfaced once backfill completed, which is precisely the behaviour the two-clock design exists to make visible.

But I have to be straight about the fix. A U-shaped score distribution is not what a real fraud or risk model produces. It is a generator implementation detail chosen to make one chart legible, and reading it as anything else would be dishonest. Every scenario parameter here, magnitudes, segment weights, sample sizes and that distribution, was tuned for demonstration. None of it is a claim about detection sensitivity on real data.

The profile I deleted

Two detection profiles ship with the project, and the difference between them is not just thresholds.

The hysteresis gap is doing as much work as the threshold. aggressive clears at 0.07 against a 0.10 fire line, a narrow band, because fast-moving adversarial inputs are expected to cross back and forth. patient clears at 0.15 against 0.25, a wide one, so a single noisy window drifting back toward baseline does not reset a slow-building trend. Separate fire and clear thresholds are what stop a statistic sitting on the line from flapping an alert open and resolved every other window.

A third profile shipped for most of the build and no longer exists. It was written for slow-moving continuous-label domains, with daily windows and regression error metrics. Once every demo scenario turned out to need a binary classification model, nothing in the repository exercised it against real data anymore. Only shallow config-loading tests touched it.

I deleted it rather than keep it as an example. An unverified profile sitting in a project about monitoring correctness is worse than not shipping one.

What this is, and what it is not

This is a systems design project. It is not a benchmark, and the evidence that matters for it is the design rather than the numbers.

That is why the synthetic data is a choice rather than a shortfall. What I wanted to demonstrate is whether the system behaves correctly under conditions I constructed deliberately: a segment failure hidden by an aggregate, a degradation invisible until backfill, a single-window spike that correctly never opens an alert. Real production traffic would not let me construct any of those on demand. Deterministic scenarios do, and a golden test regenerates every one of them twice and asserts the output is identical, so the numbers on this page can be reproduced years from now.

The honest limits are these. It has never run against real production traffic, so it has not met the specific ways real infrastructure fails at scale. It runs on a schedule over windows, so it is not a per-request detector and I would not reach for it if I needed sub-second intervention on a single transaction. Onboarding a model means writing a YAML file, which I chose because a file is reviewable and versioned and a click is not, but which does assume comfort with configuration as code. It does not remediate anything: it tells you something changed, and it does not retrain, roll back, or swap in a challenger, because I do not think a monitoring system should be trusted to make that call alone.

And the model-agnostic claim is worth stating precisely. The engine genuinely does not branch on domain: which tests run, which metrics compute, and which segments slice all come from configuration. But every scenario I have run through it is a binary classification model. The deleted profile is the proof of that gap rather than a hidden one.

Evidently, Arize and the cloud vendors all solve this commercially. I did not build this because they do not exist. I built the primitives myself because the places where monitoring actually breaks are not in the choice of statistic. They are in the bin edges, the empty buckets, the family of tests, and the gaps on a chart, and you do not learn any of those from a vendor's quickstart.

What building this changed

I came into this thinking monitoring was a measurement problem. It is mostly a problem of confidence calibration in the wrong direction.

Every serious bug in this project was a system being quietly certain when it had no grounds to be. A verifier that passed because it had nothing to check. A scan bound that was always exactly long enough because it was defined to be. A chart that drew a clean line across a three-hour scheduler outage. A p-value that reported significance at every window because significance is what p-values report once the sample is big enough. None of those look like failures. They look like healthy systems.

That is the thing worth carrying forward. A monitoring system's most dangerous output is not a wrong number. It is a calm one. So the question I now ask of any check I write is not whether it can detect the problem, but whether there is any state of the world in which it would fail. If I cannot describe that state, I have not built a check. I have built a reassurance.