Uplift Targeting

Testing when uplift modeling is worth paying for, and reporting honestly that a cheap baseline usually wins.

Executive Summary

Most models predict who will act. Almost none say who acts because you intervened, and that difference is exactly where a marketing budget is won or wasted. Chasing it is expensive, so the first question is whether it pays at all. This is a causal targeting system built on randomized-experiment data. It estimates each customer's uplift, ranks the population by it, and turns a budget into a priced who-to-treat policy. It was built as much to test whether uplift modeling earns its keep as to do it. The honest finding: plain response targeting is a hard bar. Uplift only draws level with it at scale, the edge sitting inside the fold-to-fold noise even on a million rows, and on small, clean data it loses outright. Knowing when not to reach for the complex tool is the result, and the project reports it plainly.

Project Snapshot

Project Metrics Card

No accuracy tile, and no live-latency tile: uplift has no per-row label to be accurate against, and the service was never load-tested for per-request latency (batch-scoring throughput is measured at ~67k rows/second; see System Quality). Neither missing number is faked. The headline ships with its cross-validation band because a lone Qini would misrepresent a high-variance metric. On the smaller Hillstrom dataset the honest headline is a negative: no uplift model beats the baseline, and that is stated where a metric would otherwise go.

Key Insight

The Problem

Every model in a typical portfolio answers one of two questions: what will happen or what is this. A churn model says who will leave. A fraud model says which transaction is bad. A forecast says what next quarter looks like. None of them answers the question a budget owner actually asks: what should we do about it, and to whom.

The gap matters because the people a model flags and the people an action changes are not the same people. A churn model surfaces the customers most likely to leave, but some of them will leave no matter what you do, and some will stay no matter what you do. Spending a retention offer on either group is wasted money. The customers worth targeting are the ones who stay because of the offer and would have left without it. That group is invisible to a prediction model, because "likely to churn" and "moved by the intervention" are different quantities that only sometimes overlap.

This is the space of four customer types that a response model cannot tell apart: the persuadables, who act only if treated; the sure things, who act regardless; the lost causes, who never act; and the sleeping dogs, whom the treatment actively pushes away. A budget should flow to the persuadables and avoid the sleeping dogs. A model that ranks by predicted response spends most of its budget on the sure things: the people who inflate its accuracy precisely because they were going to act anyway.

Estimating who is a persuadable requires a causal quantity: the difference between a customer's outcome if treated and the same customer's outcome if not treated. And here the problem reveals the constraint that shapes everything downstream. You can never observe both. A customer either received the treatment or did not; the other outcome is a counterfactual that never happened. Unlike a forecast, where the future eventually arrives and grades your prediction, uplift is never labeled at the level of a single person. There is no per-customer answer key, not late, not ever.

That single fact, no per-row ground truth, is the real engineering challenge, and it is not a data-collection gap that more effort could close. It is the nature of the question. Everything the system does is shaped by having to optimize, evaluate, and price a quantity it can never directly measure on any individual.

Requirements & Constraints

Functional Requirements

• Estimate a per-customer uplift score from randomized treatment/control data and rank the population by it.

• Compare several uplift models against naive baselines on one identical, cross-validated footing, and choose a winner by a written rule.

• Turn a ranking plus a budget into a targeting policy: who to treat, and the projected incremental value of doing so.

• Serve the chosen policy through an API, and drive a live demo where a budget slider updates the policy and its value.

Non-Functional Requirements

• Every headline number reproducible from a single seeded command, raw data to result.

• Free resources only (public datasets, open-source libraries, free hosting), and a small, transparent dependency set.

• The evaluation harness trusted before any model is judged by it, which means unit-tested against a reference implementation.

• Honest reporting: uncertainty shown, not hidden; a negative result published as readily as a positive one.

Hidden Constraints

The invisible reality that shaped the architecture is that the target has no per-row label, and every proxy for it is aggregate and noisy. Three consequences follow, and each one bends a design decision. Because uplift is never labeled per person, evaluation cannot be a per-row error; it has to be a ranking metric measured over groups. Because those ranking metrics (Qini and AUUC, the area under the uplift curve) are high-variance and swing with the treated-to-control balance, a single train/test split can lie in either direction, so every number has to be a cross-validated band rather than a point. And because the dollar value of a policy depends on business inputs no dataset contains, the value figure has to be a range that sweeps its assumptions, never a single confident number. The whole system is built to measure something it cannot see, and to be honest about how well it can see it.

Architecture Diagram

High-Level System Design

The pipeline runs left to right, and every stage is pure logic separated from the I/O around it, so each can be tested in isolation.

Data: Both datasets load through the scikit-uplift loader into one tidy schema with a single binary treatment, so the same code path handles a 64k-row email experiment and a million-row ad experiment. Splits are stratified on the treatment-by-outcome cells so that every fold and the hold-out preserve the arms' balance. Before any model runs, a covariate-balance check computes the standardized mean difference of each feature between arms and confirms the randomization actually holds. If the treatment weren't random, every uplift number downstream would be measuring confounding instead of effect.

Bakeoff: Eight models are fitted on the same splits: two baselines (treat everyone, and target by predicted response), three meta-learners (S, T, X) over a shared LightGBM base, and three direct uplift models (class transformation, uplift tree, uplift forest). No model is assumed best. The baselines exist to be beaten.

Evaluation: Each model is scored out-of-fold with the Qini and AUUC ranking metrics and an uplift-by-decile breakdown, all reported as bands across folds. A selection rule reads those bands, applies the beat-both-baselines test, and breaks ties on a reserved hold-out. The winner's uplift drivers are then named.

Serving: The chosen model and a slim hold-out reference are frozen into a small pickled policy bundle. A FastAPI service exposes /score, /policy, and /health; the /policy endpoint takes a budget and returns the incremental-value band with its assumptions echoed back. A Streamlit demo loads the same bundle in-process, so the live app needs no running API and no processed data at demo time: one deployable artifact.

Engineering Decisions

Six decisions shaped this system, and every one traces back to the same fact: the quantity being optimized has no per-row label, and its proxies are noisy. They are ordered as they surface in the pipeline: how to measure, what to measure against, how to trust the measurement, how to build the models, how to pick one, and how to price the result.

Decision 1: Evaluate by Ranking, Not by Per-Row Error

Context: The requirement was to judge how good each model's targeting is, but there is no per-customer uplift label to compute an error against.

Alternatives Considered:

Final Decision: The harness measures the ranking, not a point prediction. Qini and AUUC ask a question the data can answer: if you sort the population by predicted uplift and target from the top down, do the incremental outcomes actually accumulate faster than random targeting would give you. Uplift-by-decile is the readable companion: sort into ten bins and plot the observed treated-minus-control rate per bin, which should step down from the top. The trade-off accepted is that no single per-model error exists; you describe a model by a curve and a band, not a number.

Engineering Principle: Measure the decision the model actually informs. When the quantity you care about has no label, evaluate the ranking it produces, not a point estimate it can't be graded on.

Decision 2: Make the Naive Baseline the Adversary

Context: Uplift signal is thin: a difference of two noisy quantities, often tiny next to the base rate. It is entirely possible for a sophisticated uplift model to lose to a plain classifier, and easy to never notice if you don't check.

Alternatives Considered:

Final Decision: Two naive baselines are first-class competitors on every fold and metric: treat everyone (uplift constant, ranking random) and target by predicted response (a plain outcome classifier, no causal claim). Treat-everyone is really the zero line (a random ranking scores Qini 0 by construction), so the response model is the baseline that actually has to be cleared. The rule is explicit: an uplift model must beat both to earn its place, and if none does, the response baseline "wins" and that becomes the headline finding. This is what turned an anticipated negative on the small dataset into a reported result rather than a buried one.

Engineering Principle: A model is only as good as the cheapest thing it beats. Build the trivial baseline first and make it the thing to defeat; if the complex model can't, that is the finding, not an embarrassment.

Decision 3: Report Bands, Treat Single Splits With Suspicion

Context: Qini is high-variance and swings with the treated-to-control balance. On the two datasets here, a single hold-out draw disagreed with the cross-validated ranking, in both directions.

Alternatives Considered:

Final Decision: Every headline is 5-fold stratified cross-validation, seed 42, reported as mean plus standard deviation across folds. The band is the unit, not the point. The reserved 20% hold-out is scored exactly once per model and used only as a tie-break, never as the headline, because one draw of a noisy metric is precisely the thing not to trust. When the hold-out and the CV disagreed, the band settled it: a within-noise CV result cannot be overturned by a single lucky split.

Engineering Principle: Report the uncertainty as the primary quantity. On a high-variance metric, the width of the band is more honest than any point inside it, and a single split is a rumor, not evidence.

Decision 4: Hand-Roll the Methods Over LightGBM and numpy

Context: The models needed were meta-learners, uplift trees and forests, and a class-transformation estimator. Full causal-ML libraries implement all of these, but pull in heavy dependency trees and hide the mechanics.

Alternatives Considered:

Final Decision: The meta-learners, the uplift tree and forest, the class transformation, the evaluation harness, and the explainability are all hand-rolled over LightGBM and numpy. scikit-uplift is kept, but only for its dataset loaders and as a reference implementation to check the harness against, not as the thing doing the work. The cost is more code and more tests; the return is that every method on the leaderboard is transparent, and the dependency set stays small enough to install on a free tier.

Engineering Principle: Own the core logic you are judged on. A dependency is fine for the edges, but the method at the center of the project should be code you can read, test, and defend line by line.

Decision 5: Break Statistical Ties on Robustness, Not the Higher Mean

Context: On the large dataset, two models cleared the response bar on the mean: the S-learner (CV Qini +0.0927) and the uplift forest (+0.0905). The S-learner had the higher mean, but the gap (0.0022) was a fraction of either band. A paired test across the five folds puts it at p = 0.36 (95% CI [−0.008, +0.004]). They were a statistical tie.

Alternatives Considered:

Final Decision: The rule breaks ties on stability first. The forest had the tighter band (std 0.0067 vs 0.0110) and it generalized: it held +0.0936 on the hold-out while the CV-leading S-learner fell to +0.0806, the largest CV-to-hold-out drop in the field. The forest is carried into serving. The trade is recorded honestly rather than buried: the S-learner is the interpretable near-tie, directly explainable through its LightGBM internals, and a future serving choice could reasonably prefer it on those grounds.

Engineering Principle: When two results are a statistical tie, the higher mean is not a winner. Break the tie on the property that survives a new sample (stability and generalization), not on a decimal inside the noise.

Decision 6: Price the Policy as a Range, Not a Number

Context: A ranking is only half a decision. The other half is money, and money depends on two inputs no dataset contains: what a conversion is worth and what a contact costs.

Alternatives Considered:

Final Decision: The value machinery separates what it measured from what it assumed. The incremental-conversions figure comes off the conversion Qini curve and is assumption-free. The dollar value is then reported as a band that sweeps value-per-conversion ($50–$150, anchored on the data's own $116.36 average order value) against cost-per-contact ($0.05–$0.50). Every dollar figure ships next to the assumptions that produced it, and the demo's budget slider moves the whole band live.

Engineering Principle: Separate the measured from the assumed, and let the assumption travel as a range. A single confident dollar figure built on guessed inputs is the most persuasive way to be wrong.

Implementation Highlights

The instinct throughout was to make the measuring stick trustworthy before measuring anything with it, and to keep every piece independently testable.

The harness is validated before it judges. The Qini and AUUC functions are a from-scratch reimplementation, then cross-checked against scikit-uplift's own qini_auc_score and uplift_auc_score to machine precision. Only once the ruler agreed with an independent ruler was any model measured by it. This is why the leaderboard is trusted: the metric was audited, not assumed.

Evaluation math is tested first, models second. The test suite (130+ unit tests across 16 files) leads with the evaluation and data transforms, not the models. A bug in a model produces a wrong score; a bug in the metric produces a wrong decision about every model, so the metric is where the tests concentrate.

Randomization is checked, not trusted. Every dataset passes through a standardized-mean-difference balance check across arms before modeling. On randomized data the check should pass, and it does, but running it makes the assumption explicit and catches the day it wouldn't.

Reproducibility is a single seed. Everything is seeded at 42, and the same uv run command reproduces the same leaderboard, the same winner, the same figures. Dependencies are added phase by phase, each only when first used, so the environment never carries weight it doesn't need.

The serving artifact is minimal on purpose. The policy bundle is the pickled model plus a slim hold-out reference, small enough (about 6 MB, public data only) to ride into the repository so the hosted demo has everything it needs at startup. This matters because the Criteo source download is no longer available and cannot be rebuilt in the cloud.

Challenges & Debugging

Challenge: The Hold-Out Contradicted the Cross-Validation

What failed: After the cross-validated leaderboard was in hand, the single 20% hold-out told a different story on both datasets. On the small dataset, the uplift models swung above the response baseline on the hold-out (the forest posted +0.0661 against response's +0.0187), the reverse of their within-noise CV order. On the large dataset, the response baseline scored highest on the hold-out (+0.0973), edging out the very uplift models that had beaten it on CV.

Investigation: The instinct to chase might have been "which split is right." The better question was how large the swings were relative to the fold-to-fold spread. On the small dataset, every uplift model's CV standard deviation was already larger than its mean; a single 12,800-row draw of a metric that noisy is expected to swing hard in either direction.

Root cause: Not a bug. Qini is high-variance, and a single split is one sample from a wide distribution. The disagreement was the metric behaving exactly as its variance predicts.

Resolution: The band was designed to be the arbiter precisely for this case. A CV result where every uplift model sits inside the noise cannot be overturned by one lucky split, so the honest reading held: negative on the small dataset, a modest and split-sensitive positive on the large one. The hold-out disagreement went into the write-up as evidence for banding, not as a result to reconcile away.

Lesson: When a single draw disagrees with a cross-validated band, the band wins. The disagreement is not a mystery to solve; it is the reason the band exists.

Challenge: The Planned Explainer Wouldn't Build, and Wouldn't Fit

What failed: The plan was SHAP for explainability. Its numba dependency could not build against this environment's numpy under Python 3.12, and separately, the chosen model was a forest hand-rolled over numpy that no tree explainer can read anyway.

Investigation: Two problems, one honest answer. Even with SHAP installed, applying a tree explainer to a from-scratch forest was never going to work, and applying SHAP to a meta-learner's base models explains response, not the two-model difference that is the actual uplift.

Resolution: Drivers come from permutation importance: shuffle a feature, measure how far the model's Qini falls, and read the fall as how much the model leans on that feature to rank uplift. It is model-agnostic, it fits the actual winner, and it adds no new dependency. The result is readable: feature f0 dominates, hurting Qini more than twice as much as the next feature when shuffled. The honest ceiling is stated too: the features are anonymized, so this names which signal drives uplift, not what it is.

Lesson: Pick the explainer that fits the model you actually chose, not the one with the famous name. A method you can apply honestly beats a method you have to bend to make fit.

Results & Evaluation

Primary outcome is visit. Scores are out-of-fold from 5-fold stratified cross-validation (seed 42); the headline is normalized Qini on a 0–1 scale, higher is better, reported as mean ± std across folds. The hold-out column is a single unbiased number on the reserved 20%, used only to break ties.

Model Performance: the large dataset (Criteo, 1M subsample)

Only two of six uplift models beat the response baseline on cross-validation, and only on the mean. A paired test across the five folds cannot separate either from the baseline: forest minus response is +0.0028 (95% CI [−0.006, +0.012], paired t p = 0.43), S-learner minus response +0.0050 (p = 0.37). At a million rows the honest reading is a tie, not a win. The fancier meta-learners fare worst of all, and the treatment imbalance is why: Criteo is 85% treated, so the control arm the T- and X-learners lean on is thin, while the S-learner pools both arms and the forest splits on uplift directly. The forest is still the right model to carry (tightest band, best hold-out behavior, and the decile concentration below), but it is the best of a field that draws level with the cheap baseline, not one that beats it. And on the single hold-out draw the response baseline (+0.0973) actually edges the forest (+0.0936).

Model Performance: the small dataset (Hillstrom, 64k customers)

No uplift model clears the response bar on cross-validation, and every one has a standard deviation larger than its mean, statistically indistinguishable from zero (Figure 2). A paired test agrees: the forest lands a non-significant 0.0069 below the response baseline (p = 0.39). The hold-out column flips the order, but a single 12,800-row draw of a high-variance metric cannot overturn a within-noise CV result. The chosen model here is the response baseline, labeled openly as a negative: on this data, do not deploy an uplift model.

Ranking Quality: does the top of the list actually respond more?

This is the sanity check that matters more than the aggregate Qini: the model is not just scoring high, it is putting the incremental visits where the policy will actually spend the budget. About a sixfold concentration of effect in the top decile is what makes a targeting policy worth running at all.

What Drives Uplift

From a Ranking to a Priced Policy

Ranking is only half the decision; the value machinery prices it. Targeting the top 10% of the Criteo hold-out:

The incremental-conversions figure is assumption-free: it comes straight off the conversion Qini curve. The point value uses $116.36 per conversion and $0.10 per contact; the band sweeps value $50–$150 against cost $0.05–$0.50. The band crosses zero, which is the honest message: at these assumptions the policy can be net-negative, and a budget owner should see that range rather than a single reassuring number.

System Quality

The evaluation harness is a from-scratch reimplementation cross-checked against scikit-uplift to machine precision, so the leaderboard rests on an audited metric. The suite runs 130+ unit tests, weighted toward the evaluation math. The randomization balance check passes on both datasets. The full run reproduces from one seeded command in under four minutes of wall time. Scoring the chosen forest is cheap: about 67,000 rows per second in a single process, so the 200,000-row Criteo hold-out scores in roughly three seconds and even the full ~14M-row Criteo would batch-score in a few minutes; per-row scoring is embarrassingly parallel. Per-request serving latency under load was not benchmarked.

After Deployment

Everything above is an offline study, and uplift carries a deployment twist that offline work never faces. The evaluation here rests on random assignment: treatment was assigned by coin flip, and that is the one fact that makes a treated-minus-control difference an unbiased uplift estimate. Deploying the policy ends it. The moment you start targeting by predicted uplift, assignment is no longer random. It is correlated with the very features that drive the outcome, so the next batch of production data is confounded, and the Qini harness that graded every model here would be measuring the policy's own selection, not incremental effect. A causal targeting system can quietly stop being measurable the day it goes live.

The fix needs no new architecture, only a discipline: hold back a small randomized slice of traffic that stays randomly assigned no matter what the policy says: a learning budget. Uplift stays estimable on that slice, and the exact cross-validation harness in this repo re-runs on it on a schedule. The cost is a few percent of budget spent non-optimally; the return is that the model stays gradeable against the same honest bar it was chosen by.

Three health signals fall out of what is already built. The balance check (standardized mean difference across arms) runs on the randomized slice to confirm assignment is still random. The score distribution is watched for drift against the hold-out reference already frozen in the bundle. And realized top-decile uplift on the randomized slice is the metric that actually matters: if the roughly sixfold concentration in Figure 5 decays toward the population average, the ranking has gone stale and retraining fires; absent a signal, retrain on a fixed cadence.

Rollback is already cheap because the unit of deployment is deliberately small. The policy bundle is one immutable pickled file (model plus hold-out reference), so versioning is keeping the previous bundles and rollback is swapping the file back, with no schema migration and no data rebuild. What this project did not do is build that online loop or load-test the service; it ships the artifact and the offline projection, and names the randomized-slice loop as the first thing a real deployment would need. That is the honest boundary: the study shows the model can be measured well, and is explicit that operating it is the next system, not this one.

Business Impact

The value here is a decision framework and an honest boundary around it, not a booked return. The datasets are anonymized ad and marketing data, so this is a methodology demonstration, not a deployed retention system.

• It targets incrementality, not response. Ranking by uplift sends the budget to the persuadables (the people an action actually moves) instead of the sure things a response model rewards for converting anyway. The demo's decile chart shows a roughly sixfold concentration of real effect in the top slice.

• The money it reports is a range, not a promise. A budget produces an incremental-value band under stated cost and value assumptions, so a decision-maker sees the downside case (the top-10% Criteo band runs from −$8,811 to $2,567) rather than a single figure engineered to look safe.

• The negative is the most useful finding. "Response targeting is a cheap, hard bar that uplift, at best, only draws level with, and only at scale" tells a budget owner when not to pay for a complex model. On the small dataset, the right call is to skip uplift modeling entirely, and knowing that before spending on it is worth real money.

Limitations

Leading with these, because they are where the project earns its seniority.

• No per-row ground truth. You never see both outcomes for the same person, so uplift is never labeled at the row level. There is no per-customer error to report; all evaluation is aggregate. This is the nature of the problem, not a gap in the build.

• Aggregate metrics are noisy. Qini and AUUC have high variance and swing with the treated-to-control balance. Every headline is a cross-validated band, and the CV-versus-hold-out disagreement on both datasets is the direct evidence for why.

• The honest negative is real. On the small dataset, uplift modeling does not beat response targeting. That finding is reported, not buried.

• Bounded validity, and deployment ends the randomization. Estimates speak only to these populations and these exact randomized treatments. Acting on the policy replaces random assignment with targeted assignment, the very thing that makes the uplift estimate unbiased, so a live system has to hold back a randomized slice to stay measurable (see After Deployment). Any projected value is an offline projection, never a promise.

• Two datasets, many differences. The "uplift needs scale" reading rests on two datasets that differ in size and domain, base rate, and treatment strength at once. Scale is the leading explanation for why uplift helped on Criteo and not Hillstrom, but with n = 2 it is a hypothesis, not a proven cause. The full-Criteo run in Future Improvements is the direct test.

• The dollar figure rests on assumptions. Value-per-conversion and cost-per-contact are inputs, not measurements, so incremental value ships as a range. Anonymized features cap interpretability at "which feature," not "what it means."

Lessons Learned

Lesson 1: Measure the Decision, Not the Metric the Data Can Afford

The temptation with any model is to reach for the familiar metric. Here the familiar metric, a per-row error, does not exist, and the next-most-familiar, response AUC, answers the wrong question. Choosing Qini and uplift-by-decile meant scoring the ranking the policy actually uses. Generalization: the right metric is the one that mirrors the operational decision, even when it is noisier and harder to summarize than the one out of habit.

Lesson 2: An Honest Negative Is a Senior Result

A model that quietly loses to a baseline, dressed up as a win, is the failure mode this project was built to avoid. Making the naive baselines first-class competitors turned "uplift didn't help on the small data" from an embarrassment into the headline. Generalization: the credibility of a positive result is set by how visibly you would have reported the negative one.

Lesson 3: On a Noisy Metric, the Band Is the Result

Every disagreement between the cross-validation and the hold-out resolved the same way: trust the band, distrust the single draw. Designing the band in from the start meant no lucky split could rewrite the story. Generalization: report uncertainty as the primary quantity; a point estimate on a high-variance metric is a headline waiting to be wrong.

Lesson 4: Own the Logic You Are Judged On

Hand-rolling the models and the harness cost more code and more tests, but it meant every number on the leaderboard came from code that could be read and defended, and that the ruler could be audited against an independent one. Generalization: use dependencies at the edges, but the method at the center of the work should be yours to inspect.

Lesson 5: Complexity Must Clear a Cheap Baseline Before It Earns Deployment

Uplift modeling is more complex than response targeting in every way: to fit, to evaluate, to explain. It drew level with response targeting only on the large dataset, and even there the edge did not survive a paired significance test. Two datasets is thin evidence, and they differ in more than size, so "scale" is the leading read, not a proven one. Generalization: sophistication is a cost, not a virtue; it has to beat the cheap default on the metric that matters before it is worth shipping, and sometimes, even at a million rows, it doesn't.

Future Improvements

• Observational causal inference (propensity weighting and backdoor adjustment) to lift the randomized-data restriction and handle data where the treatment wasn't assigned at random.

• Multi-arm and continuous treatments, and the R-learner left optional in v1, to move past a single binary treatment.

• A full-Criteo run over all ~14M rows to tighten the confidence bands beyond what the 1M subsample allows, the most direct way to firm up a split-sensitive result.

• Uplift on the rarer conversion outcome, not just visit, where the signal is thinner and the value machinery matters more.

Technology Stack

References

• GitHub: github.com/i-hridaysaha/uplift-targeting

• Full evaluation write-up: reports/phase8_report.md in the repository (leaderboards, hold-out, drivers, all figures)

• Data: Hillstrom MineThatData Email Challenge; Criteo Uplift Prediction dataset; both loaded via scikit-uplift