Fraud Detection Platform
Payment fraud hides in under 1% of transactions, and every alert costs a human analyst's time. This platform is a from-scratch recreation of a real-time fraud system I built at AXA. I rebuilt it around a sharper framing: fraud is not a classification problem but a ranking problem, bounded by how many alerts a team can review in a day. That one shift moves the decision threshold, not raw model accuracy, to the center of the product.
Project Snapshot
Scale / Complexity | Core Technique | Engineering Challenge | Differentiator |
435 RPS, p99 27.8 ms, single core | XGBoost + streaming features | Leakage & delayed labels | Train/serve parity, gated retrain |
Key Insight
Fraud detection is not a classification problem but a ranking problem constrained by human review capacity — which moves the decision threshold, not model accuracy, to the center of the system.
Architecture Diagram
The Problem
Every payment processor faces the same asymmetry. Fraud is rare, typically one to three transactions in a thousand, but each missed fraud is a chargeback, and each false alarm is a legitimate customer declined at the till or a fraud analyst spending minutes on a clean transaction. The visible task is "detect fraud." The task that actually shapes the system is different.
Start with the naive framing: train a classifier, measure accuracy, ship the best one. On a dataset where 99.8% of transactions are legitimate, a model that predicts "not fraud" every single time scores 99.8% accuracy. It is useless, and accuracy cannot tell you so. The first constraint appears before a single feature is built: the metric everyone reaches for is actively misleading here.
The deeper constraint is human. A fraud team can only review so many alerts per day. A model that flags 70% of fraud sounds impressive until you see it flags twenty legitimate transactions for every real one, a review queue no team can staff. The real question is never "how accurate is the model" but "given that analysts can review N transactions a day, how much fraud can the model surface inside that budget." That reframing changes what "good" means, and everything downstream follows from it.
Two more realities complicate this. Fraud labels arrive late: a transaction is only confirmed fraudulent when a chargeback or investigation lands, days or weeks later, so the model is scoring today against ground truth it won't have until next month. And fraud is adversarial: patterns that separated fraud from legitimate traffic last quarter degrade as tactics shift. The system isn't detecting a fixed thing; it's tracking a moving one with delayed feedback.
None of these are solved by a better algorithm. They are solved by designing the pipeline around them.
Requirements & Constraints
Functional Requirements
Score a transaction in real time and return a fraud probability plus a human-readable explanation.
Compare multiple model families on one identical, chronologically held-out test set.
Expose a tunable decision threshold so the business can trade precision against recall.
Monitor the deployed model for both input drift and performance decay.
Non-Functional Requirements
Sub-100 ms scoring latency at payment-authorization time.
Throughput at 1M+ transactions per day.
Per-transaction explainability, treated as a compliance requirement in financial services, not a feature.
No data leakage anywhere in the training or feature pipeline.
Hidden Constraints
The invisible reality that shaped the architecture is that the model never operates alone: it feeds a fixed-capacity human review process, against ground truth that arrives weeks late. That single fact demotes accuracy, promotes the decision threshold to a first-class tunable, and forces monitoring to split into two separate problems because the signals that reveal trouble don't arrive at the same time.
High-Level System Design
The pipeline runs in two planes: an offline training plane that produces a model and a threshold, and an online serving plane that scores live traffic.
Offline: Raw transactions are split chronologically, earlier data training and later data testing, before any feature touches them. Features are then computed causally, using only information available strictly before each transaction's timestamp. Three model families train on the same split under a configurable imbalance strategy. The winning model's threshold is tuned on a validation slice, and SHAP (Shapley additive explanations) produces both a global feature-importance report for model governance and a per-transaction explainer for analysts.
Online: A FastAPI service exposes POST /score. It returns a probability, applies the tuned threshold, and, when asked, attaches the top contributing factors for that transaction. Velocity and device-history features are not recomputed per request; they are read from a real-time feature store that maintains per-customer state incrementally. The API loads its model from the registry by stage (whatever is currently in Production), not from a hardcoded artifact path, resolved once at startup, so a fresh promotion is picked up on the next worker reload rather than mid-process.
Continuous: A monitoring service watches the input distribution immediately and the model's realized precision and recall only once delayed labels arrive, two jobs on two clocks, and feeds a gated retraining loop that promotes a new model only when it beats the incumbent.
Engineering Decisions
Eight decisions shaped this system, and each traces back to the same three facts: a rare event, a capped review queue, and labels that arrive weeks late. They are ordered as they surface in the pipeline — metric first, then the data split, the imbalance correction, the model, the threshold, monitoring, serving, and finally the lifecycle that keeps it current.
Decision 1: Optimize the Operational Metric, Not Accuracy
Context: The requirement was to pick "the best model," but on a <1% base rate the default metric silently rewards a model that does nothing.
Alternatives Considered:
Option | Advantages | Disadvantages |
Accuracy | Familiar, one number | Rewards the all-negative predictor; blind to the rare class |
ROC-AUC | Threshold-independent, standard | Optimistic under heavy imbalance; dominated by the easy majority |
PR-AUC + tuned threshold | Focuses on the rare positive class; maps to review capacity | Requires choosing an operating point deliberately |
Final Decision: The pipeline reports precision, recall, F1, and PR-AUC, and never accuracy. PR-AUC was chosen as the primary comparison metric because precision and recall are exactly the two quantities a fraud team negotiates: precision is how much analyst time is wasted, recall is how much fraud escapes. The trade-off accepted is that no single number summarizes the model, and you must state an operating point to describe its behavior.
Engineering Principle: Choose the metric that mirrors the real operational decision; a metric that can be maximized by doing nothing is measuring the wrong thing.
Decision 2: Chronological Splits and Strictly Causal Features
Context: Fraud features are temporal: spending velocity, device history, merchant reputation. Those temporal features are where leakage hides.
Alternatives Considered:
Option | Advantages | Disadvantages |
Random train/test split | Simple, standard in tutorials | Leaks the future: rolling features see transactions that hadn't happened yet |
Time-based split, naive features | Chronologically honest | Still leaks if a feature aggregates across the whole dataset |
Time-based split, causal features | No future information anywhere | More care per feature; slower to compute |
Final Decision: Every split is chronological and every feature is computed using only data strictly before the transaction's timestamp. Merchant risk scores are target-encoded on the training split alone. A first-ever transaction gets a large sentinel for "time since last transaction," never zero, because zero would falsely imply rapid repeat activity. The cost is that features can't be vectorized as casually, since each one has to respect a moving cutoff.
Engineering Principle: In any system with time, the split and the features must both obey the arrow of time; a leak isn't a bug you find in testing. It's a number that looks great and lies.
Decision 3: Reweight the Loss Instead of Resampling the Data
Context: With fraud at a fraction of a percent, a model trained on raw class proportions barely notices the positive class.
Alternatives Considered:
Option | Advantages | Disadvantages |
No correction | Honest baseline | Model under-weights fraud; poor recall |
SMOTE oversampling | Balances classes; can help recall | Fabricates minority points; easy to leak if fit before the split |
Class weighting (scale_pos_weight) | Touches the loss, not the data; no synthetic points | Doesn't create new signal, only reweights existing |
Final Decision: Class weighting is the default; SMOTE and no-correction remain switchable for comparison. Weighting was preferred because it reweights the existing loss rather than inventing minority examples, and because SMOTE, if fit before the chronological split, would leak synthetic points derived from validation and test fraud back into training. When SMOTE is used, it is applied strictly after the split. The accepted trade-off: weighting can't manufacture signal that isn't there.
Engineering Principle: Prefer the correction that changes the least about your data; every synthetic sample is a small assumption you now have to defend.
Decision 4: XGBoost as the Production Model
Context: The model must be accurate enough at a fraction of a percent base rate, fast enough for sub-100 ms scoring, and explainable enough to satisfy governance.
Alternatives Considered:
Option | Advantages | Disadvantages |
Logistic Regression | Fast, transparent, great sanity check | Underfits nonlinear fraud patterns |
Random Forest | Strong nonlinear baseline; good recall | Higher latency; false-positive rate too high untuned |
XGBoost | Strong accuracy/latency trade-off; native imbalance support; exact TreeExplainer SHAP | More hyperparameters to reason about |
Final Decision: All three train on the same held-out chronological test set; XGBoost serves production. It won on the combination of three properties, not on any one: the accuracy-to-latency ratio at high volume, native scale_pos_weight support for imbalance, and first-class SHAP TreeExplainer support that yields exact, fast attributions rather than approximations, which matters when every alert must be explainable to an analyst and an auditor. Logistic Regression stays as a signal sanity check; Random Forest as a feature-importance cross-check.
Engineering Principle: Model selection is a constraint-satisfaction problem, not a leaderboard; the model that wins is the one that clears latency, imbalance, and explainability at once.
Decision 5: Tune the Threshold to Analyst Capacity
Context: A trained model outputs a score; the business decides at what score to act. That decision, not the model, sets the false-positive load.
Alternatives Considered:
Option | Advantages | Disadvantages |
Fixed 0.5 cutoff | Default, requires no tuning | Arbitrary under imbalance; ignores review budget |
F1-optimal threshold | Balances precision and recall automatically | May exceed the team's review capacity |
Precision-constrained (high-recall) | Caps the false-positive rate to a staffable level, then catches all the fraud it can inside that budget | Needs a precision floor chosen from staffing, not from an equation |
Final Decision: The threshold is tuned on the validation set only, and the pipeline reports both the F1-optimal point and a precision-constrained high-recall alternative. This is deliberate: a real fraud team picks its operating point from its staffing, not from an equation. Exposing two points lets the business choose "catch the most fraud we can review" over "maximize a statistic." The trade-off is explicit and owned: higher recall means more analyst hours.
Engineering Principle: The decision threshold is a business lever, not a model detail; ship it as a tunable and let the people who staff the queue set it.
Decision 6: Split Monitoring Into Two Problems on Two Clocks
Context: In production the model can fail in two ways that look similar but arrive at different times.
Alternatives Considered:
Option | Advantages | Disadvantages |
Monitor accuracy only | Simple dashboard | Can't compute it, labels arrive weeks late |
Monitor input drift only | Available immediately, no labels needed | Drift ≠ decay; a shifted input isn't always a worse model |
Separate drift and decay tracking | Early warning now, ground truth later | Two pipelines, two schedules |
Final Decision: Monitoring is split. Population Stability Index tracks feature and prediction-score drift immediately, with no labels required (PSI under 0.1 stable, 0.1–0.25 investigate, above 0.25 significant). A separate scheduled job recomputes precision, recall, and F1 only once delayed labels (chargebacks, confirmed reports) land for a past batch. Realized decay is the primary retraining trigger, because input drift alone doesn't prove the model got worse; sustained significant drift is kept as a secondary, early trigger: a shift that persists across several cycles can justify a pre-emptive retrain before labels confirm the damage. The two alerts stay distinct because they mean different things and demand different responses.
Engineering Principle: When ground truth is delayed, monitor the earliest reliable proxy for warning and the true metric for action, and never confuse the two.
Decision 7: Compute Serving Features Online, and Prove They Match Training
Context: The temporal features (velocity over trailing windows, device novelty, time since last transaction) depend on a customer's recent history. The batch pipeline computes them by scanning the whole dataset, which is correct for training and impossible at authorization time, when there is one transaction and milliseconds to score it.
Alternatives Considered:
Option | Advantages | Disadvantages |
Recompute from history per request | Reuses batch code exactly | Requires the customer's full history in-request; far too slow |
Trust the caller to supply features | Simple API | Pushes the hard problem elsewhere; nothing guarantees the numbers are right |
Stateful online store, parity-tested | Millisecond reads; features stay causal | A second implementation that must not drift from training |
Final Decision: A real-time feature store maintains per-customer state incrementally — sliding-window velocity via Redis sorted sets, a device set, a last-transaction marker — with a pure-Python fallback so the system runs without Redis. It follows a two-phase causal contract: get_features reads state as of now, excluding the current transaction, then commit folds that transaction in for the next one. This ordering reproduces the batch semantics exactly: velocity excludes the current row, and device novelty is judged against prior devices only.
That equivalence is the whole risk. A second feature implementation can silently disagree with training, so a parity test runs the same transactions through both paths and asserts the feature rows match. The accepted trade-off is two implementations behind one guardrail, because the online path cannot be a batch scan and the offline path cannot be a stream.
Engineering Principle: The most dangerous production bug in ML is train/serve skew; if you must compute a feature two ways, make their equality an automated test, not an assumption.
Decision 8: Version the Model, and Gate Every Promotion
Context: A model that ships as a mutable file path answers "what's in production?" with a filename and no memory of the data, config, or metrics that produced it, which turns the first misbehaving model into an archaeology project instead of a one-line rollback.
Alternatives Considered:
Option | Advantages | Disadvantages |
Overwrite a joblib file | Trivial | No history, no rollback, no lineage |
Version artifacts by hand | Full control | Reinvents a registry, poorly |
MLflow registry with stages | Lineage, stages, rollback for free | A dependency; stage vocabulary to learn |
Final Decision: Every training run logs its params, metrics, tuned threshold, feature list, encoders, and a fingerprint of the training data to a local MLflow registry, then registers the model under a Staging or Production stage. The API serves whatever is in Production, resolved by stage.
Retraining is automated but gated. A decay signal trains a challenger on the recent data window through the identical leakage-safe pipeline, and the challenger is promoted only if it beats the champion on PR-AUC by a configured margin on a common held-out split; otherwise it is logged and rejected. The decision itself records why it fired, so every promotion is auditable. The trade-off is more moving parts, in exchange for a system that can never silently ship a worse model and can always roll back to a known-good one.
Engineering Principle: Automating retraining without a promotion gate automates the shipping of worse models; the gate, not the retrain, is the feature.
Implementation Highlights
Configuration lives in one place (src/config.py), so imbalance strategy, thresholds, and drift bands are tuned without touching pipeline logic. The imbalance strategy is a single command-line switch (class_weight, smote, none), which turns "does this correction actually help" into a one-flag experiment rather than a code change. The none baseline exists specifically to measure how much the correction earns.
The serving layer draws a hard line between what it computes and what it reads. Velocity and device-history features need historical context, so they live in the online store as incrementally-updated state rather than being reconstructed per request. A stateless scorer cannot hold state, so the state lives beside it. Explainability is wired into the same response path: pass explain: true and /score returns the top contributing factors for that transaction, which is exactly what an analyst's triage screen needs.
Tests target the two failure modes that matter most here. Leakage: rather than asserting on accuracy, tests assert that features respect their temporal cutoff, because a leak passes every accuracy check and fails only in production. Train/serve skew: a parity test asserts the online store and the batch pipeline produce identical features for the same stream. And the promotion gate is tested in both directions: it must promote a better challenger and reject a worse one, because a gate only tested on the happy path is not a gate.
Results & Evaluation
All numbers below are on synthetic data whose fraud signal is a deliberately modest approximation of the real distribution, so absolute precision is capped by construction. What should be read here is the shape of the trade-offs and the discipline of the pipeline, not the magnitude of the scores. Run the same code on the real benchmark dataset and separation improves, because genuine fraud leaves a far stronger signal than this synthetic stand-in.
Model | ROC-AUC | PR-AUC | Precision | Recall | F1 |
Logistic Regression | 0.542 | 0.101 | 0.026 | 0.463 | 0.050 |
Random Forest | 0.757 | 0.136 | 0.048 | 0.733 | 0.091 |
XGBoost (F1-optimal threshold) | 0.722 | 0.109 | 0.067 | 0.262 | 0.106 |
XGBoost (default 0.5 threshold) | 0.722 | 0.109 | 0.051 | 0.554 | 0.094 |
The table tells the project's story better than any single score. Random Forest catches ~73% of fraud (recall 0.733) but at a precision of 0.048, twenty false alarms per real catch, a queue no team can staff. XGBoost gives the best PR-AUC/F1 balance, and its two rows show the threshold at work: the same model swings from 0.55 recall to 0.26 as the cutoff moves, trading fraud caught for analyst time saved. That swing is the product. Choosing where to sit on it is the operational decision the whole system was built to expose.
SHAP confirms the model leans on the features the design intended (velocity, merchant risk, device novelty) rather than on artifacts, which is the global validation a governance review would demand before trusting it.
Lifecycle in Action: a Simulated Decay Event
Absolute model quality on synthetic data is weak by construction, so the lifecycle machinery is proven a different way: by injecting a clearly-labelled simulated concept-drift event (fraud moves onto different PCA components, old ones scrambled, amounts inflated) into a recent data window and watching the loop react end to end. A single run of the demo produced:
Stage | Signal | Outcome |
Drift check | 7 features in the significant-PSI band; score PSI 0.195 (moderate), the feature PSIs drive the alert | DRIFT alert |
Delayed-label check | realized F1 0.041 (floor 0.05); recall 0.27 | DECAY alert |
should_retrain | decay + sustained drift | fires, critical, with reasons ("F1 41% below baseline…") |
Gate, accept | champion PR-AUC 0.018 vs challenger 0.951 | challenger promoted to Production |
Gate, reject | champion PR-AUC 0.986 vs a stale challenger 0.019 | challenger refused, champion kept |
Rollback | promote a prior version | Production returns to the earlier model |
The two gate rows are the point: the same gate that promotes a genuinely better challenger refuses an inferior one, on real trained models, never auto-shipping a worse model, and always able to roll back. Every number here is from an executed, seed-pinned run (scripts/simulate_decay_demo.py, seed=42) and reproduces byte-for-byte; the drift is labelled simulated because it is.
System Performance
Model quality is capped by synthetic data; serving performance is not, and it was measured rather than asserted. A load test drove 20,000 store-backed /score requests (each scoring and committing state) on a single laptop core, after warming the store with 10,000 transactions across 2,000 customers:
Metric | Value |
Throughput | 435 RPS |
Latency p50 | 18.1 ms |
Latency p95 | 24.2 ms |
Latency p99 | 27.8 ms |
Errors | 0 / 20,000 |
The sub-100 ms target holds with room to spare on one core. A concurrency sweep also recorded where it breaks: throughput plateaus near 440 RPS because a single Python worker, held to one core by the global interpreter lock (GIL), does CPU-bound inference, and past ~12 concurrent requests the queue in front of that worker dominates latency (p99 blows out to 1.7 s at concurrency 64). That saturation point is reported deliberately: knowing where a system fails is worth more than a single flattering number, and the horizontal-scaling answer (stateless workers behind the shared store) follows directly from where the bottleneck sits.
Limitations
Data: The synthetic generator's fraud/legitimate separation is simplified; real validation requires confirmed investigation outcomes, not PR-AUC on a held-out synthetic set. The reported model-quality metrics should be read as a demonstration of methodology, not of production accuracy. This is the one gap the code cannot close. It is a data problem, not an engineering one.
Serving scale, measured single-node: The latency numbers are real but from one uvicorn worker with the in-memory feature store, because the test box had no Redis available. The Redis path is implemented and parity-tested but not itself load-tested at scale; the horizontal story (stateless workers over shared Redis) is designed, not benchmarked.
Tuning: No hyperparameter search is included. Fixed, reasonable XGBoost defaults are used. Optuna or Ray Tune would come before treating any model as final.
Registry currency: The registry pins MLflow 2.x for its stage vocabulary; MLflow 3 replaces stages with aliases, so a migration is owed before this tracks upstream.
Gate comparison encoding: The champion/challenger gate scores both models on a common test set encoded with the challenger's feature encoders (the merchant-risk map fit on the challenger's window), not a from-scratch re-featurization per model. Both models see the identical matrix, so the ranking between them is fair; a fuller setup would re-featurize the common test through each model's own encoders. Flagged in code, not hidden.
Promotion propagation: The serving API resolves the Production-stage model at startup, so a gate-approved promotion or rollback takes effect on the next worker reload, not instantly in a running process. Zero-downtime hot-swap (a registry watcher or a reload signal) is designed, not implemented.
Governance: No formal model card or bias audit across customer segments is included, which a regulated deployment would require.
Lessons Learned
Lesson 1: The metric is a design decision, not a reporting step
Context: accuracy silently rewarded the do-nothing model. Principle: pick the metric that mirrors the operational decision before choosing the model. Generalization: in any rare-event system, the evaluation metric determines the architecture as much as the algorithm does.
Lesson 2: Time discipline outranks model choice
Context: a random split would have leaked future information into every velocity feature. Principle: the split and the features must both obey the arrow of time. Generalization: leakage is the most expensive bug in temporal ML because it improves your metrics and hides until production.
Lesson 3: The threshold is where engineering meets the business
Context: the same model was staffable or unstaffable depending only on its cutoff. Principle: expose the operating point as a first-class lever owned by the people who staff the queue. Generalization: many ML "accuracy" debates are really unspoken disagreements about the operating point.
Lesson 4: Delayed feedback splits monitoring in two
Context: the true metric arrives weeks after the transaction it judges. Principle: monitor the earliest reliable proxy for warning, the true metric for action. Generalization: whenever ground truth lags, an early proxy signal is the difference between noticing failure and discovering it.
Future Improvements
Distributed feature aggregation: Move the online store's window maintenance behind Flink/Kafka Streams so it survives multi-region traffic, and load-test the Redis path at target scale.
Hyperparameter search: Add Optuna with a time-aware cross-validation scheme.
Online threshold adaptation: Let the operating point adjust to observed analyst review volume instead of being fixed at training time.
Shadow scoring: Score a promoted challenger on live traffic without acting on it, to confirm the gate's offline verdict holds online before it takes over.
Model card and segment bias audit: Add before any regulated deployment.
Technology Stack
Category | Technologies |
ML | XGBoost, scikit-learn, imbalanced-learn (SMOTE), SHAP |
Backend | FastAPI, Uvicorn |
Data | pandas, NumPy |
Feature store | Redis (sorted sets), in-memory fallback |
Registry / lifecycle | MLflow (local file backend), champion/challenger gate |
Monitoring | PSI drift, APScheduler, delayed-label performance job |
Load testing | Locust / asyncio + httpx |
Testing | pytest (leakage, train/serve parity, promotion gate) |
Definition of Done
What problem was solved? Ranking rare fraud so a fixed-capacity team catches the most it can actually review — without leaking the future or trusting a misleading metric.
Why this architecture? The model never acts alone; it feeds humans against labels that arrive weeks late. That promotes the threshold and the split, not the algorithm, to the decisions that matter.
What should the reader remember? Optimize for the operational decision, not the academic metric. The constraint that stays invisible in development is the one that dominates production.