Fraud Detection Platform
A fraud model can look brilliant in testing by using information that only exists after a payment is decided, then fail once live. This platform scores each transaction from its past alone, and measures how much shortcuts flatter the score.
Executive Summary
Bottom line: The shipped model scores 0.5451 PR-AUC (0.5274 to 0.5614); a random split alone would have reported 0.8044, and none of that gain exists at authorisation.
Business problem: fraud is 3.50 percent of transactions, the label is a chargeback that lands weeks later, and the endpoint has milliseconds and only the card's past.
What was built: every split, encoder and feature reading only the past, an online store held equal to training by a test, and a gated retrain loop.
Result, stated honestly: test PR-AUC 0.5451 (0.5274 to 0.5614) on public data, never deployed; the split is nine tenths of what all four shortcuts together would add.
Reading time: about 27 minutes, counting every visible word including tables and captions, or 34 with the four deep-dive sections opened.
GitHub: https://github.com/i-hridaysaha/fraud-detection-platform
Live demo: https://i-hridaysaha.github.io/fraud-detection-platform/demo/, a self-contained replay of the lifecycle run built from the committed artifacts and served from docs/demo/index.html; there is no hosted scoring endpoint.
Project Snapshot
Dimension | Detail |
Scale and complexity | 590,540 transactions, 434 columns |
Core technique | Point-in-time gradient boosting |
Engineering challenge | Future-free features at authorisation |
Differentiator | Leakage priced, not assumed |
Project Metrics Card
Key Insight
The most important lesson was that the techniques scoring highest on this data all read the future, so every feature, encoder and split had to be held to what an authorisation endpoint knows, and the difference measured rather than assumed.
Jump links
Architecture Diagram
The Problem
A card transaction arrives, and the system has milliseconds and whatever it already knows about the card to approve it, hold it for review, or decline it. On the public IEEE-CIS stream this project runs on, 20,663 of 590,540 transactions over 182 relative days are fraud, 3.50 percent; in the held-out test month, 3,083 rows in 88,581, so approving everything is 96.5 percent accurate and catches nothing.
The endpoint needs a score now, from the state it holds at that instant; the review team needs a queue it can work; neither has the label, because a fraud label is a chargeback that matures weeks later.
The approaches that score highest on this dataset do so for a reason unrelated to authorisation. A competition hands over the whole test set at the start, so a statistic over every row of a card, rows not yet happened included, is available when scoring, and a random split puts a card's later transactions in the training set, so the model learns the card rather than the fraud. Those are correct uses of what the task provides; the task differs from an endpoint, and the model measured here must be one that could have run at the moment each transaction happened, reading everything before the cut Figure 1 draws and nothing after it.
The visible problem is detecting fraud; the real one is that the best-known scores here are earned with information the endpoint does not have.
Requirements and Constraints
Functional Requirements
Score one transaction at a time: from the per-key state held at that instant, folding the transaction in only after the score is out.
Decide, not just score: approve, review or block from stated costs on a calibrated probability, with the review volume reported.
Replace the model on evidence only: a challenger ships when its paired advantage clears a margin derived from measured noise; a rollback moves the alias back.
Non-Functional Requirements
No fitted object sees a later row: a runtime guard raises, not warns.
Online equals batch: the store's feature vector is bit-identical to the training row, asserted on both backends.
Every number regenerable: one artifact, one command, a bootstrap interval; the raw data never committed.
Hidden Constraints
The dataset has no customer. Labels cluster on whatever real thing was compromised, and the card field alone is not that thing: if it were, every transaction sharing a value would share the day the card began, and only 15.4 percent of its multi-transaction values do. The card field plus the billing region plus that start day gives 217,850 entities at a label purity of 96.6 percent, against 84.8 percent for the card field alone. That key is the card as far as this page can see; every aggregate, the store and every interval marked "by card" are keyed on it. Under it 66.0 percent of test rows sit on entities the training window never saw, at a fraud rate of 4.40 percent against 1.70 percent on entities it had.
That turnover is what a random split hides (Figure 4): 74.1 percent of its test rows share an entity with training, against 34.0 percent chronologically, so the model is asked about cards it has already seen. The same model on the same features scores +0.2593 higher under it, the largest single leak on this page; Results prices it beside the other three.
Late labels are the other half. If open labels are read as legitimate, the model learns that the newest rows are never fraud, and every served row is a newest row: with the last 60 days of training labels still open, the fraud rate the model implies on the test month falls to 0.04 percent against an observed 3.48 percent.
Two thirds of test rows sit on entities training never saw and labels arrive late; a random split hides the first, worth +0.2593 alone.
High-Level System Design
The stream is split at fixed timestamps into a training window of 413,378 rows over days 1 to 120 and a validation and a test month of 88,581 rows each, and a guard raises on any fit that sees a later row. Preparation is fitted on the training window alone; nulls stay nulls, because 251 of the 372 columns with nulls carry the label in their missingness, and target encodings read labels 14 days behind the row.
The point-in-time feature build computes each row from rows strictly before it, 36 features over three keyspaces, and the parity run (Figure 5) covers 40 once four graph counters added later are included. The shipped model reads none of them: the ablation found the block added nothing beside the provided counters, the file's own C block, whose construction is the vendor's.
They are built and served anyway because the serving path is written for a stack that reads them: a bundle whose column list names a store feature reads it from the store's vector, a test covers that join, and a challenger that reads the block is served by the same path under the same parity guarantee. The guarantee is on the feature vector, not on the shipped score; what that score does rest on, the provided counters, is under Limitations.
The three planes in Figure 3 meet at the alias. Training compares three families under three imbalance strategies, ablates twelve stacks over three seeds, sets the threshold and band edges, and registers the bundle under the alias production. The service loads the alias, reads the store, prepares, scores, calibrates, bands, commits and returns the top SHAP factors. The drift job compares each column's population stability index (PSI) with the training reference at each batch close and reads no labels; the decay job scores PR-AUC once labels mature; the gate is where they meet, and it moves the alias.
One rule runs through every plane: nothing reads a row after its own, and where the rule cannot be structural it is a test.
Engineering Decisions
Decision 1: Split chronologically, test month untouched
Context
The rule starts at the split. Labels cluster by entity at 96.6 percent purity, so a split that scatters an entity across train and test is a memorisation shortcut, and one that trains on later rows measures interpolation.
Alternatives Considered
Option | Advantages | Disadvantages |
Random 70/15/15 split | The library default; the narrowest intervals; the highest number | Trains on the future and scatters entities: 74.1 percent of test rows share an entity with training, and the score sits +0.2593 above the chronological one |
GroupKFold on the entity key | Removes entity leakage exactly | Still mixes time, and every evaluation entity has zero history, where the chronological test month has 34.0 percent of its rows on seen entities |
Chronological 70/15/15 at fixed timestamps | Reproduces the production ordering; entity overlap lands where the data puts it; drift is measurable by window | Two thirds of evaluation rows are cold entities, and the intervals are wider |
Final Decision
Why this option won: it asks the question the endpoint will ask, about entities and weeks the model has never seen.
Which trade-offs were accepted: lower, noisier numbers, and a test month spent once, so selection runs on validation.
Which assumptions were made: the 0.70 and 0.85 timestamp quantiles, leaving two evaluation windows of about a month.
Engineering Principle
A split is a claim about what production will ask; choose it for that, not for the interval it produces.
The chronological split asks about entities and weeks the model never saw, which is the endpoint's question and the reason the number is lower.
Decision 2: Lag the label encoding, exclude the entity
Context
A chronological split protects the rows; it does not protect the labels the encoders read. A target encoding replaces a category with a summary of its labels, and out-of-fold encoding is not enough on a chronological problem: a row on day 100 has no label on day 100. At 96.6 percent purity, the entity key's summary is nearly a lookup of its own label.
Alternatives Considered
Option | Advantages | Disadvantages |
Out-of-fold encoding, no lag | Stops a row reading its own label | Does nothing about timing: the train-to-validation gap of +0.0812 AUC is the signature of labels not yet arrived |
A conservative lag by convention, 30 or 60 days | Safe | Unmeasured; a lag is a cost as well as a protection, and the sweep shows the cost |
Sweep the lag, choose the smallest whose gap is indistinguishable from zero | Measured: the gap falls to +0.0077 at 14 days, inside its half width of 0.0173 across the 18 identifier columns encoded, and to -0.0068 at 28 | Validation AUC alone would pick lag 0, so the rule had to be stated before the sweep was read |
Encode the entity key too, on its validation AUC of 0.7835 | A strong number that holds at every lag | Split on entity overlap it is 0.9877 on rows whose entity training saw and 0.4915 on the rest: a lookup and a constant averaged over two base rates |
Final Decision
Why this option won: the gap, not the score, is the leak signature, and 14 days is the smallest lag whose gap sits inside its own half width; the entity decomposition shows a shape no lag repairs.
Which trade-offs were accepted: the encoding is undefined for the first 14 training days, and a served row reads the tables at its own day minus 14, tables that end with the training window; the sweep prices a longer effective lag at 0.0117 of validation AUC for 28 days against 0.0068 for 14, both against lag 0.
Which assumptions were made: the lag is not a claim about how long a chargeback takes; on a stream the tables hold only labels that have arrived, so the effective lag is the longer of 14 days and the chargeback delay, which this file cannot measure.
Engineering Principle
Write the acceptance rule before the sweep runs, and make it ask for a shape, not a threshold on the headline.
A lagged encoding closes a +0.0812 gap, and an entity encoding at 0.7835 was a label lookup wearing a feature's clothes.
Decision 3: Every window strictly before its own row
Context
Labels were one way to read the future; the features are the other. The best-known construction on this dataset attaches an entity's mean of the provided counters to every row of the entity, its first included; at authorisation a card's first transaction has no later rows to average. A per-key trailing window is the point-in-time version; the question was where the exclusion of the present row lives.
Alternatives Considered
Option | Advantages | Disadvantages |
Ship the published aggregation | Carried the first-place competition entry; 76 extra columns | A function of rows that have not happened yet, with no online form |
Ship it with a label lag | Reuses the machinery of Decision 2 | A lag protects against labels not yet arrived, not rows not yet happened; lagged, it is no longer the published feature |
Build it behind a switch for measurement | Convenient comparison | Puts an undeployable construction one boolean from the serving path; the measurement lives in a script outside the package |
Trailing windows by searchsorted over the time-sorted stream, side="left" on both bounds | The exclusion is arithmetic, not a keyword; ties are excluded by construction; per-key prefix sums and Welford moments are the state a store holds | Ties cost signal: simultaneous transactions see each other one transaction later |
Final Decision
Why this option won: a rolling join is right with closed="left" and wrong by one word otherwise; two side="left" arguments with no other purpose cannot drift quietly.
Which trade-offs were accepted: the shipped model reads none of the 36 causal features, which moved validation PR-AUC by -0.0052, +0.0029 and -0.0076 over three seeds beside the provided counters; they stay built because the serving path and its parity test are written for a stack that reads them, and the shipped stack is not that stack.
Which assumptions were made: that the price of deployability should be measured: +0.0371 (+0.0290 to +0.0458) on this window.
Engineering Principle
Put a correctness rule in arithmetic a test can break, not in a parameter a tidy-up can flip.
The published aggregation is worth +0.0371 here and cannot exist at authorisation, so it stays out of the production path at any price.
Decision 4: Read, then commit, in two calls
Context
The batch build's exclusion is arithmetic; a store that holds state per key and answers one transaction at a time has to keep the same "strictly before t", and its failure is silent: fold the transaction in before reading, and every number looks right and disagrees with training by exactly one transaction.
Alternatives Considered
Option | Advantages | Disadvantages |
One call that reads the state and updates it | The shape most feature-store clients have | The order of the two operations is invisible at the call site, so a refactor can flip it with no caller changing |
One call with a commit flag | Same call, order configurable | The flag's default becomes the contract, and a caller who omits it gets whatever the default encodes |
Two calls, get_features then commit, never one that does both | A read writes nothing and a commit returns nothing, so the wrong order is two visible lines; testable directly | Every caller carries two lines; the guarantee needs per-key ordered arrival, and an out-of-order row is refused |
Final Decision
Why this option won: the property is testable from outside: two reads of one transaction agree, and a read at the latest committed timestamp sees the state before it.
Which trade-offs were accepted: a Redis commit costs 0.90 ms of round trips against 0.039 in process, and a transaction older than its key's latest committed one is refused with a 409 rather than folded in.
Which assumptions were made: per-key ordered arrival, which a deployment would take from a partitioned queue; and that bit-for-bit equality was reachable, which it was, since the store subtracts the same prefix sums.
Engineering Principle
Make the dangerous ordering visible at the call site, then test the property, not the implementation.
Over 134,339 rows and 40 features the store and batch build differ by 0.0, and each was broken once to prove the test sees it.
Decision 5: Band edges from a stated cost matrix
Context
With the score now the same online as in training, what remains is turning it into a decision. A fraud operation has a review team, so it needs two edges, not one threshold, and a reason for where they sit. The file carries no chargeback amount, margin or analyst cost, so the reason can only be stated and kept apart from the arithmetic.
Alternatives Considered
Option | Advantages | Disadvantages |
One threshold tuned for F1 on validation | Needed anyway to compare models at one operating point | No review band, and F1 weighs a missed fraud and a false alarm equally, which no operation does |
Two thresholds set by alert volume | Speaks the review team's language | Volumes with no derivation behind them, moving with traffic rather than with risk |
Two thresholds derived from a cost matrix | Three costs stated as assumptions, two edges by arithmetic, the volumes and realised cost reported afterwards | The costs are guesses by construction, and the matrix has no capacity term |
Final Decision
Why this option won: with a missed fraud at 10, a false decline at 1 and a review at 0.5, review beats approve above a calibrated probability of 0.05 and block beats review above 0.5; change a cost and the edges move.
Which trade-offs were accepted: a review queue of 331.0 rows a day on the test month, and whether a team can work that is not a question this data answers.
Which assumptions were made: the three costs, and that a review resolves the transaction correctly.
Engineering Principle
Keep assumed inputs and derived outputs in separate places, so nobody quotes the arithmetic as the assumption.
Three stated costs give two edges by arithmetic, so the review queue is a consequence a business can change by changing a cost.
Implementation Highlights
The decisions above set the rules; this is where each is enforced, and where the textbook defaults were measured before being kept or dropped.
Go deeper: guard, schema, drop rules, drift edges and routes
The guard raises: assert_train_only refuses a frame with a row at or after the boundary, a frame with no timestamp column, an empty frame and a null timestamp, so it cannot pass vacuously. It was replaced with a pass-through once, and the suite failed.
Structure, not distribution: the schema declares the column set, dtype families, nullability on four columns and sign bounds from the measured data, and deliberately no upper bounds, because 130 of 393 numeric columns move a median or a 99th percentile by more than 10 percent between windows and a cap fitted on train would reject legitimate later rows. It caught its own first bug: fourteen standardised block means failed a "no negative observed" rule inherited from raw columns.
Four named drop rules: 434 columns to 288, with 14 dropped as effectively constant and 132 as redundant; the two rules a pipeline written on intuition would lean on hardest, the near-empty columns and the time-inconsistent ones, dropped nothing, because every near-empty column carries the label in its missingness and all 59 flagged columns held up once fitted on the whole window.
A transformation measured and refused: normalising the 15 provided timedelta columns (the D block) to an origin day raised drift on all 15, because the raw deltas were already stationary and the origin writes the calendar into the column.
One thread, measured: the booster is 0.081 ms of a 37.6 ms request, and pinning it to one thread is right for a single row (1.49 to 1.59 times faster) and wrong for a batch of a thousand.
Drift edges calibrated on the accepted windows: the textbook PSI edge of 0.20 puts 18.4 columns a week in alert on a model that is fine; the per-column edge, the worst in-control week held out one at a time, gives a false alert rate of 0.0 on 10 weeks. The 16 target encodings in the shipped stack are reported and never alert, because their served distribution is one point of the path the training window was encoded along.
What an operator sees: three routes. POST /score returns the score, the calibrated probability, the band and the top factors; GET /model names the loaded model, alias, version and run; GET /health adds the store backend, the inference thread count and the requests served. A row out of order for its key is a 409 with the reason, a payload the schema refuses is a 422 listing every violation rather than the first.
The demo is the artifacts: docs/demo/index.html is built from the committed artifacts, and a test compares the committed page with a fresh build byte for byte.
The schema caught its own first bug, the guard was broken on purpose and the suite failed, and three textbook moves were measured and refused.
Challenges and Debugging
Challenge: the repeat-purchase feature that reversed
What failed: rows repeating an earlier purchase on six fields ran at 1.333 times the fraud rate of the rest on the training window, and the feature was written up as shipping.
Lesson: a relationship measured on one window is a candidate, not a feature; the ship rule now asks for the same sign on all three splits.
Go deeper: how the reversal was traced and what shipped instead
Investigation: the per-split coverage table, read for an unrelated reason, showed the marked rows at 0.814 and 0.883 times the unmarked rate on validation and test, with the Wilson intervals disjoint on every split and in opposite directions to train.
Root cause: six of ten recency deciles sit below the base rate and hold most of the marked population; an identical purchase within three minutes runs at about 3.4 times the rate of a row with no history, one from months back below it, and an all-history flag averages a strong positive over a large mild negative whose mix is not stationary.
Resolution: the count restricted to the last hour holds on every split at 2.865, 2.410 and 2.019, and ships with the recency as a continuous column; the all-history count stays as a labelled candidate.
The project's own features were held to the same rule as the published shortcuts, and one failed it before it shipped.
Results and Evaluation
Every number here, and every tile in Figure 2, comes from one configuration: chronological 70/15/15, a test month of 88,581 rows over days 152 to 182, seed 42 for the headline and three seeds for the ablations, thresholds tuned on validation and applied unchanged, every interval a 95 percent bootstrap over 1,000 test-month resamples, by row unless marked by card, paired where two models share rows.
What each shortcut would have bought
The model is held at the shipped configuration and one shortcut at a time is switched on; Figure 6 prices each against it.
Variant | Columns | Test PR-AUC (95 percent interval) | Against the baseline | Reading |
Causal baseline (the shipped configuration) | 186 | 0.5451 (0.5274 to 0.5614) | reference | reproduces the shipped model |
(a) Encoders fitted on every row | 186 | 0.5511 (0.5336 to 0.5678) | +0.0060 (+0.0013 to +0.0108) | inflates by row; by card the interval covers zero |
(b) Entity aggregates over every row | 262 | 0.5822 (0.5648 to 0.5981) | +0.0371 (+0.0290 to +0.0458) | inflates |
(c) Entity-mean post-processing of the scores | 186 | 0.4898 (0.4711 to 0.5070) | -0.0554 (-0.0656 to -0.0463) | deflates |
(d) Random split instead of chronological | 186 | 0.8044 (0.7926 to 0.8171) | +0.2593 (+0.2378 to +0.2809) | inflates |
(a) + (b) + (c), chronological | 262 | 0.5190 (0.5011 to 0.5360) | -0.0262 (-0.0399 to -0.0141) | deflates by row; by card the interval covers zero |
(a) + (b) + (c) + (d), random | 262 | 0.8317 (0.8203 to 0.8439) | +0.2866 (+0.2653 to +0.3092) | inflates |
Two of the four did not inflate by card, and both were expected to: (a) is +0.0060 by row (+0.0013 to +0.0108) and +0.0060 by card (-0.0084 to +0.0214), where the resample draws whole entities, and (c) deflates. The post-processing hurts because purity counts entities and PR-AUC counts rows: the 457 test entities that mix labels hold a third of the fraud, most in the null-key bucket. Without the split, the competition-style pipeline is worse than the causal one.
What late labels do
At 15 days of open labels (Figure 7) the immature model loses 0.0345 (0.0268 to 0.0428) of test PR-AUC and the model fitted without that window loses 0.0124 (0.0059 to 0.0190); at 60 days the immature model implies a fraud rate of 0.04 percent and raises 0.3 alerts a day against the reference's 78.2. Dropping the window costs less than keeping its labels at every N; the true delay is not established, so no N is recommended.
Which model, and which stack
Nine fits on one 237-column stack (Figure 8): XGBoost without reweighting leads on validation at 0.6198, ahead of class weighting at 0.5745 and SMOTE at 0.5373, and the ordering holds in every family; reweighting moves the threshold, not the ranking.
The shipped stack drops 51 columns and sits at -0.0039 (-0.0084 to +0.0008) against the default one, +0.0328 (+0.0241 to +0.0418) over the best forest. By card its interval widens to 0.5100 to 0.5805, and that width is where the promotion margin comes from.
At the tuned threshold of 0.2519 it reads precision 0.6034, recall 0.4713 and 78.2 alerts a day (75.1 to 81.2) on test, at a median 37.6 ms per request on one thread; the serving tail and the worker scaling are in this section's deep dive.
The gate on a drifted stream
A drift injected into the replayed months (Figure 9) raised the drift flag at cycle 3, from the rows alone, and the decay flag at cycle 8, once labels matured under an assumed 30-day window.
A decay flag needs two things from a weekly batch (8,111 to 21,273 rows): a drop of at least 0.06 against the acceptance reference, and a drop interval that excludes zero. On ten in-control weeks four drops excluded zero, three of them improvements; the one loss, 0.0510 (0.0091 to 0.0904), was refused by the tolerance alone, by 0.0090. The weekly half widths run 0.0352 to 0.0572, so a tolerance derived on the month clears the widest week's own half width by less than 0.003, and that headroom is thin.
Each challenger is fitted on the trailing 60 matured days, calibrated on the next 14 and gated on the 14 after that by a paired bootstrap by card. Of four, two were promoted at +0.0954 and +0.0621, one was refused as not better, and one was refused at +0.0424 with its interval above zero but under the margin. A rollback moves the alias one step back, and the served path reproduces the batch path on it. None of this machinery books a return on its own; what it is worth depends on costs the file does not carry.
Go deeper: label latency, leaderboard, bands and the serving tail
Labels still open for the last N days | Immature: test PR-AUC (against the reference) | Immature: implied fraud rate | Excluded: test PR-AUC (against the reference) |
|---|---|---|---|
15 | 0.5106 (-0.0345 (-0.0428 to -0.0268)) | 2.99 percent | 0.5327 (-0.0124 (-0.0190 to -0.0059)) |
30 | 0.4593 (-0.0858 (-0.0961 to -0.0754)) | 1.95 percent | 0.4877 (-0.0574 (-0.0662 to -0.0493)) |
45 | 0.4163 (-0.1289 (-0.1413 to -0.1170)) | 1.15 percent | 0.4839 (-0.0613 (-0.0706 to -0.0528)) |
60 | 0.3435 (-0.2017 (-0.2163 to -0.1881)) | 0.04 percent | 0.4732 (-0.0719 (-0.0825 to -0.0616)) |
90 | 0.1629 (-0.3823 (-0.3964 to -0.3664)) | 0.00 percent | 0.4493 (-0.0959 (-0.1069 to -0.0855)) |
Family | Imbalance | Columns | Validation PR-AUC | Test PR-AUC (95 percent interval) | Test ROC-AUC |
|---|---|---|---|---|---|
Logistic regression | none | 606 | 0.4094 | 0.3655 (0.3476 to 0.3818) | 0.8464 |
Logistic regression | class weight | 606 | 0.3862 | 0.3414 (0.3233 to 0.3577) | 0.8506 |
Logistic regression | SMOTE | 606 | 0.3892 | 0.3480 (0.3295 to 0.3646) | 0.8476 |
Random forest | none | 237 | 0.5530 | 0.4744 (0.4573 to 0.4914) | 0.8907 |
Random forest | class weight | 237 | 0.5466 | 0.5123 (0.4940 to 0.5299) | 0.9027 |
Random forest | SMOTE | 237 | 0.5134 | 0.4804 (0.4617 to 0.4983) | 0.8778 |
XGBoost | none | 237 | 0.6198 | 0.5490 (0.5316 to 0.5662) | 0.9111 |
XGBoost | class weight | 237 | 0.5745 | 0.5221 (0.5044 to 0.5398) | 0.8975 |
XGBoost | SMOTE | 237 | 0.5373 | 0.4998 (0.4822 to 0.5180) | 0.8748 |
XGBoost, shipped stack | none | 186 | 0.6184 | 0.5451 (0.5274 to 0.5614) | 0.9050 |
Band on the test month | Rows per day | Share of all fraud | Fraud rate in band |
Approve, calibrated probability below 0.05 | 2,500.1 | 25.3 percent | 1.01 percent |
Review, 0.05 to 0.5 | 331.0 | 38.3 percent | 11.6 percent |
Block, 0.5 and above | 46.9 | 36.4 percent | 77.7 percent |
Serving, one worker, a serial load client of 2,000 requests: the server's p50 was 37.4 ms, p95 39.7 and p99 83.5, with no errors. The tail is the one-row frame, not the model: over five timed repeats its p99 is 77.2 ms against a p50 of 4.6, preparation next at 29.2 against 23.8, and the booster's p99 is 0.2. Four workers reach 62.1 requests per second at a concurrency of 8, p99 213.2 ms, no errors, against 25.2 on one.
The adversarial validation that separates the training window from the test month reaches an out-of-fold AUC of 1.0000 on the shipped columns, 0.9031 without the target encodings and 0.8600 without the identifiers, their encodings and the day counters: the windows differ by population turnover and the calendar, not by what fraud looks like. The drift monitor is quiet on that same turnover by construction: each column's edge is the larger of the textbook 0.20 and the worst of the ten in-control weeks, strictly above, so it watches for a move beyond what the accepted windows already showed rather than for the turnover itself. The random search over expanding-window folds scored 0.6048 on the full validation window against the default's 0.6184, so the default configuration ships.
The honest score is 0.5451; the split alone would have reported 0.8044, and the machinery that keeps the difference visible is the deliverable.
Business Impact
Never deployed, so no return is booked; the product is a decision framework and the measured cost of getting the discipline wrong.
The decision it enables: three bands from stated costs, so the review queue, 331.0 rows a day (324.9 to 337.3) on the test month at the cost-derived edges, follows from costs a business writes down.
What the discipline is worth: under the stated matrix the bands realise 0.1492 (0.1429 to 0.1558) per row against 0.3480 (0.3352 to 0.3598) for approving everything; a row's value is an input, so the return is a ratio, not a figure.
What the shortcut would cost: a team promoting on a random-split number would expect 0.8044 and receive 0.5451 (0.5274 to 0.5614), and could not tell that from decay.
What late labels cost: open labels read as legitimate cut the alerts at the tuned threshold from 78.2 a day to 0.3, a silence a fraud team would read as success.
The value here is knowing which offline number to believe, and that knowledge was measured rather than assumed.
Limitations
An independent build on public IEEE-CIS data: no employer data, code or number appears on this page or in the repository. The table names what that data cannot establish.
Category | What remains unsolved | Why | What would be required |
Data | No merchant signal, no observable multi-market structure | No column names a merchant, and the billing-country field is one value on 99.0 percent of rows | A seller identifier and a country column that varies; until then signal strength is not comparable with a production system's |
Inputs | The shipped model reads the provided counters, timedeltas and engineered columns, whose construction belongs to Vesta, the data's provider | Nothing fetched for the repository establishes what the provided counters (the C block) count or whether they are point-in-time; the built velocity correlates with them at most 0.2773, and the fraud rate where the strongest one is positive is 4.525 times the rate where it is zero, against 2.556 for the strongest built feature | A definition from the data owner, or an upstream system that computes them at authorisation; until then the point-in-time rule is proven for the project's own features and assumed for the provided ones |
Evaluation | Card-level labels, a simulated label delay, and a test month the shipped columns tell apart from training | Labels cluster by entity at 96.6 percent purity, the file carries final labels with no chargeback dates, so the latency sweep is a deterministic cutoff, and an adversarial classifier separates the two windows at AUC 1.0000 on population turnover and the calendar | A measured chargeback delay, a customer identifier that is not reconstructed, and a longer stream than six months |
Deployment | Single-machine replay, never deployed | One worker saturates at 25.2 requests per second and four at 62.1; per-key ordering comes from the load client and a late arrival is refused; the store backend is fixed at startup and its behaviour under a Redis outage is not established; every number is from one laptop | A partitioned queue, a late-arrival policy with a bounded tolerance, an outage policy for the store, a row-native preparation path, a run on another machine |
Assumptions | The cost matrix, the maturity window and the gate margin | The costs of 10, 1 and 0.5 and the 30-day maturity are chosen inputs; the 0.06 margin comes from measured noise but was exercised near its edge once; the decay reference ratchets down with no floor | Real costs, a real delay, a derived floor |
Every number here holds on one dataset; the ones that would change on a real stream are named above, not hidden.
Lessons Learned
Lesson 1: A metric earned with information you will not have at decision time is not a result
Context: the split alone moved the score by more than every feature-level shortcut combined.
Engineering principle: price the shortcut before believing the number; a gain that touches no feature is a property of the question, not the model.
Generalization: an offline evaluation is a claim about what production will ask; check it by asking that question.
Lesson 2: Put the rule in arithmetic and tests, not in conventions
Context: the present-row exclusion is two side="left" arguments, the store's ordering is two calls, the guard raises; each was broken on purpose.
Engineering principle: a rule in a keyword default or a contributing note is one refactor from gone.
Generalization: where a property cannot be structural, make it a test that has been seen failing.
Lesson 3: Write the acceptance rule before reading the number
Context: the entity encoding at 0.7835 passed a threshold rule and failed the shape rule; one ablation rule was rewritten after the numbers, and both drafts are in the artifact.
Engineering principle: a rule chosen after the result launders a preference into a measurement.
Generalization: record the rule, the number, and the order they happened in.
Future Improvements
Go deeper: six changes a real stream forces, and the stack
A maturation window from a measured delay: the sweep says the cheapest window costs 0.0124 and the immature alternative 0.0345; a chargeback date column would turn that into a recommendation.
An absolute decay floor: the reference ratchets down on promotion, and the model at the end of the demo runs at six tenths of the level the shipped one was accepted at with nothing in the lifecycle knowing it.
Per-segment edges shipped, not only reported: on the segment whose base rate moved between months, its own calibrator put 207 rows in review where the global one put 1,297.
A row-native preparation path: the batch pipeline reused for one row is correct by construction and 37.6 ms by construction, and the p99 tail sits in building that one-row frame; a second implementation would need its own parity check.
A partitioned queue: per-key ordering is what the two-call contract assumes, and the load client supplies it today.
A hosted endpoint: a live score can be added without touching the demo page, which was built to show the loop rather than one number.
Technology Stack
Category | Technologies |
Machine learning | XGBoost, LightGBM (probes), scikit-learn, SHAP |
Data | pandas, NumPy, Parquet caches |
Serving | FastAPI, Redis sorted sets with an in-process fallback |
Registry and lifecycle | MLflow registry aliases, a challenger fit and gate in the package |
Quality | pytest with artifact tests in CI, ruff, mypy strict on src/, an em dash check, a pre-push hook |
Documentation | a numbers script per document, a verification scan of every numeric token, a static demo page built from the artifacts |
References
Resource | Link |
|---|---|
GitHub | |
Live demo | https://i-hridaysaha.github.io/fraud-detection-platform/demo/, built from docs/demo/index.html |
Documentation | README.md, METHODOLOGY.md, docs/adr/, docs/verification.md |
Dataset | Kaggle competition ieee-fraud-detection, provided by Vesta; not redistributed |
Prior work | the first-place construction, NVIDIA Technical Blog, McDonald and Deotte, 26 January 2021, quoted in ADR 0020 |