Uplift Targeting
A model that predicts who will respond also picks customers who would have responded anyway. This project ranks customers by how much a contact changes their behaviour, prices a budget against that ranking, and reports when the simpler model wins.
Executive Summary
Bottom line: The simpler model held at 64,000 customers, drew level with uplift at a million rows, and lost, narrowly, at fourteen million: the deployment decision.
Business problem: contact budgets should go to customers a contact moves, but no customer shows both outcomes, so that effect is never labelled per person.
What was built: six uplift models against two naive baselines, eight in all, on one cross-validated harness, a written selection rule, and a priced budget step.
Result, stated honestly: no uplift model beat the response model on 64,000 customers; on a million rows the forest edged it inside both fold bands and trailed it on the hold-out; on all fourteen million both uplift models cleared it on every fold, by four to six percent of its Qini.
Reading time: about 33 minutes, counting every visible word including tables and captions, or 41 with the four deep-dive sections opened.
Live demo: https://uplift-targeting-zpatkwkjfwqvnqdxpu9rzc.streamlit.app, a Streamlit app that scores the untouched hold-out live and reprices the policy from a budget slider.
Project Snapshot
Dimension | Detail |
|---|---|
Scale and complexity | 1.06M rows, eight models |
Core technique | Uplift meta-learners and forests |
Engineering challenge | No per-row ground truth |
Differentiator | Honest negative, priced policy |
Project Metrics Card
Public, anonymised, randomised email and ad data; never deployed. Qini is normalised and reported as mean ± std over five folds; the dollar band sweeps $50 to $150 per conversion against $0.05 to $0.50 per contact.
Key Insight
The most important lesson was that uplift is never labelled per customer, so models had to be judged as rankings, in cross-validated bands, against a cheap baseline the system was built to let win.
Jump links
Architecture Diagram
The Problem
A churn model names who will leave and a forecast names next quarter. Neither answers the question a budget owner asks: what should we do, and to whom. The customers a model flags and the customers an action changes are not the same people. Someone who would have visited anyway is wasted budget, someone a nudge tips over is real gain, and a few people a nudge pushes away are worse than doing nothing. A model that ranks by predicted response spends most of the budget on the first group, the people who inflate its accuracy because they were going to act regardless.
Both experiments show an effect that is real and uneven. In the Hillstrom email test 16.71 percent of treated customers visited against 10.62 percent of control, a lift of 6.09 points; in the Criteo ad test the lift is 1.08 points. Sliced, it moves: across Hillstrom's prior-spend deciles it climbs from 5.04 points to 7.84, and in Criteo the lowest decile of one anonymised feature carries 6.38 points against roughly zero elsewhere. An average hides that; targeting is the act of finding it.
The quantity that finds it is causal: a customer's outcome if contacted minus the same customer's outcome if not. Nobody observes both. Each customer received the contact or did not, and the other outcome never happened, so unlike a forecast, which the future eventually grades, uplift is never labelled for any single person. There is no answer key, not late, not ever.
The obvious build models that effect directly and ranks by it. Six such models were built. The question this page answers is not whether they work but whether any of them earns a budget over the plain classifier it was meant to replace.
Whom to contact is the visible problem; the real one is an effect with no per-row label, so every model is judged in aggregate.
Requirements and Constraints
Functional Requirements
Estimate and rank: a per-customer uplift score from randomised treatment and control data, and a population ranked by it.
Compare on one footing: several uplift models against two naive baselines, treating everyone (the zero line) and a plain response classifier (the bar), on identical folds, with the winner chosen by a rule written before the numbers.
Price the ranking: a budget, a count or a fraction becomes a who-to-treat policy with a projected incremental value.
Non-Functional Requirements
Seeded end to end: every headline reproduces at seed 42 from the persisted splits, one command per phase; Criteo's raw archive has to be supplied locally, because its public download is dead.
Free and small: public datasets, open-source libraries, free hosting, and dependencies added only when a phase first needs them.
The ruler audited first: the evaluation harness is tested against a reference implementation before any model is judged by it.
Hidden Constraints
The target has no per-row label, and every proxy for it is aggregate and noisy. Three consequences follow, and each bends a decision below.
Evaluation is a ranking metric, not a per-row error: it has to be read over groups, which is what Qini, the incremental outcome a ranking captures above random targeting, and AUUC, the area under the uplift curve, are. Decision 1 builds that.
Every number is a band: those metrics are high-variance and depend by construction on the treated-to-control balance, 2 to 1 on Hillstrom and 85 to 15 on Criteo, so a single split can rank a lucky model first. Decision 3 makes the band the unit.
The price is a swept range: what a conversion is worth and what a contact costs are inputs no dataset carries. Decision 6 carries them as ranges.
Figure 4 is the constraint made visible. On Hillstrom's 51,200 development rows every uplift model's fold-to-fold spread is wider than its own mean, so none can be told from zero, while the response model sits at +0.0197 ± 0.0170. A six-point effect on a 14.68 percent base rate is, on this many rows, too thin to rank.
No label, noisy proxies, and a price built on assumptions: so the system reports rankings in bands against a baseline it must let win.
High-Level System Design
The pipeline in Figure 3 runs top to bottom, every stage pure logic kept apart from the I/O around it. Both datasets load through the scikit-uplift loader, Criteo from a local archive because its public downloads stopped answering. They land in one schema with a single binary treatment: Hillstrom's two email arms collapse to one treated flag, and Criteo's randomised flag is used as it comes, its post-treatment exposure flag ignored. Splits stratify on treatment by outcome so every fold and the 20 percent hold-out keep the arm balance and the base rate. Before any model runs, a standardised-mean-difference check per covariate confirms the randomisation on both datasets.
Eight models then fit on the same folds: treat-everyone and a response model as baselines; three meta-learners over a shared LightGBM base, S, one model with treatment as a feature, T, one model per arm, and X, T's two models blended by propensity; and three direct models, a propensity-corrected class transformation, an uplift tree and a bagged uplift forest. Each is scored out of fold on Qini, AUUC and uplift by decile as mean and standard deviation across folds, and the five per-fold values are kept beside the band.
A selection rule applies the beat-the-bar test on the cross-validated mean, breaks ties among the survivors on the tighter band, and reads the hold-out once, as confirmation rather than as a vote. The shipped ranking and the baseline are then read at the top 10 percent of the hold-out, where the policy spends. Permutation importance names the winner's drivers.
The chosen model, its feature schema and a slim hold-out reference freeze into one pickled policy bundle, stamped with the lightgbm and numpy versions that built it so a later dependency bump warns instead of moving a score silently. A FastAPI service reads it for /score, /policy and /health; the Streamlit demo loads the same bundle in process, so the hosted app needs no running API and no processed data, and on Hillstrom it adds readable segment filters over the reference rows. Each stage embeds one of the six decisions that follow.
The measuring stick is built before the models it judges, and the bundle it selects is the only artifact that ships.
Engineering Decisions
Decision 1: Evaluate the ranking, not a per-row error
Context
There is no per-customer uplift label to compute an error against, and the two datasets differ in base rate, 14.68 against 4.72 percent, and in arm ratio, so their scores also have to sit on one scale.
Alternatives Considered
Option | Advantages | Disadvantages |
|---|---|---|
Per-row uplift error | Familiar, one number per model | Impossible: the label does not exist for any customer |
Response classifier AUC | Easy, well understood | Measures who responds, not who is moved, which is the wrong question |
Normalised Qini and AUUC with uplift by decile | Scores the ranking the policy will use; the 0 to 1 scale compares datasets with different base rates and arm ratios | Aggregate and high-variance, so it needs a band to read |
Final Decision
Why this option won: Qini asks what the data can answer: sorted by predicted uplift and treated from the top, do incremental visits accumulate faster than random targeting? The size-corrected curve, which scales the control arm up to the treated arm's size, handles Criteo's 85 to 15 arms; the decile view is the readable companion.
Which trade-offs were accepted: no single per-model error; a model is a curve and a band. A whole-curve coefficient can also rank two models by the part of the curve no budget ever reaches, so Results reads the curve at the operating point as well.
Which assumptions were made: the coefficients are defined to equal scikit-uplift's qini_auc_score and uplift_auc_score, and the from-scratch implementation is tested equal to them to machine precision; the reference library is the audit, not the engine.
Engineering Principle
When the quantity has no label, measure the decision it informs: the ranking, not a point estimate that cannot be graded.
The harness scores the ranking the policy will act on, and it was tested equal to an independent implementation before it judged anything.
Decision 2: Make the naive baseline the adversary
Context
A ranking metric needs a reference. Uplift is a difference of two noisy rates, often small next to the base rate, so a sophisticated model can lose to a plain classifier without anyone noticing. Two references are cheap: treat-everyone, a constant score whose Qini is the diagonal at exactly 0.0000, and a response model, one LightGBM classifier of the outcome fitted on the whole population with the treatment column dropped.
Alternatives Considered
Option | Advantages | Disadvantages |
|---|---|---|
One uplift model, its Qini reported | A simple story | No reference; a mediocre number looks fine alone |
Uplift models compared only to each other | A winner emerges | The winner may still be worse than doing something trivial |
Every model against treat-everyone and the response model on the same folds | A real bar, and the negative case is exposed | More to build, and a negative headline is possible |
Final Decision
Why this option won: treat-everyone is the zero line and the response model is the bar. An uplift model must clear both on the cross-validated mean to be considered at all; the band rule in Decision 3 decides among those that do. The asymmetry is deliberate: the baseline is the default, not a competitor, so a lead on the mean is the entry ticket, not the verdict, and if none clears the bar the response model wins by default.
Which trade-offs were accepted: the bakeoff can end in a negative, and on Hillstrom it did. A point test at the bar also admits a lead inside the noise, and on Criteo it did, which is why every Criteo claim here carries its hold-out qualifier and the deployment case is made at the operating point, not on the coefficient.
Which assumptions were made: the response model shares its base-learner parameters with the meta-learners, so the comparison is between learners, not tuning budgets.
Engineering Principle
A model is only as good as the cheapest thing it beats; build that first and make it the thing to defeat.
Treat-everyone is the zero line, the response model is the bar, and the rule lets the bar win.
Decision 3: Report bands, score the hold-out once
Context
With the bar set, the question is how many samples of the metric to trust. Qini moves with the split; on both datasets the single hold-out disagreed with the cross-validated order.
Alternatives Considered
Option | Advantages | Disadvantages |
|---|---|---|
One train and test split | Standard, one number per model | One draw of a high-variance metric can rank a lucky model first |
Cross-validation, the mean reported | Averages some noise | A lone mean hides how wide the spread is |
Five stratified folds, mean ± std, hold-out scored once to confirm | Shows the uncertainty; a single draw cannot overturn it | Five fits per model, and a two-column result to explain |
Final Decision
Why this option won: every headline is five-fold stratified cross-validation at seed 42, mean and standard deviation across folds, with the standard deviation the unit, and the five per-fold values persisted beside it. The 20 percent hold-out, 12,800 rows on Hillstrom and 200,000 on Criteo, is scored once per model and read as confirmation of the pick; it never sets the headline, and as coded it is not a vote in the rule.
Which trade-offs were accepted: five fits per model, and a headline that is a band rather than a number.
Which assumptions were made: a gap smaller than the larger of the two bands is a tie. That is a one-standard-deviation heuristic, not a test, and the paired test in the Results deep-dive agrees with it: no pair on either dataset separates.
Engineering Principle
On a high-variance metric the width of the band is the result, and a single split is a rumour.
Every Qini on this page is a band across five folds; the hold-out is scored once, read as confirmation, never promoted to the headline.
Decision 4: Hand-roll the methods over LightGBM and NumPy
Context
Three meta-learners, an uplift tree and forest, and a class transformation were needed. scikit-uplift 0.5.1 ships no tree or forest, and the causal libraries that do carry heavy builds on Python 3.12.
Alternatives Considered
Option | Advantages | Disadvantages |
|---|---|---|
causalml or econml wholesale | Every method ready | Heavy dependency trees with build risk on Python 3.12, and the method being judged is a black box |
scikit-uplift's own models | Already installed | No tree or forest at all, and its class transformation assumes 50/50 arms, which is biased on 2 to 1 and 85 to 15 data |
Hand-roll every method over LightGBM and NumPy | A small dependency set, and every method readable and unit-tested | More code, and the tree needed its own proof before it could join the board |
Final Decision
Why this option won: the meta-learners are thin wrappers over the LightGBM already installed. The class transformation regresses the propensity-corrected outcome Y·(T − p)/(p·(1 − p)), tested unbiased against the plain variant on unequal arms. The tree splits histogram-binned features on a Euclidean divergence criterion with a per-arm minimum in every leaf; the forest bags 50 such trees with a random feature subset per split. Both joined the leaderboard only after recovering an effect set in synthetic data.
Which trade-offs were accepted: no TreeSHAP for the forest, which the second challenge pays for, and a 150,000-row bootstrap cap per tree on Criteo to keep pure-NumPy fits fast.
Which assumptions were made: on randomised data the propensity is the constant treated fraction, which the X-learner blend and the transformation both use.
Engineering Principle
Own the logic you are judged on; a library method encodes assumptions, and the one checked here encoded the wrong arm ratio.
Every method on the leaderboard is code that can be read and tested, and the library shortcut checked would have been biased on both datasets.
Decision 5: Break the tie by the rule written first, not the higher mean
Context
On Criteo two models cleared the bar on the cross-validated mean: the S-learner at +0.0927 ± 0.0110 and the uplift forest at +0.0905 ± 0.0067. The gap, 0.0022, is a fraction of either band.
Alternatives Considered
Option | Advantages | Disadvantages |
|---|---|---|
The higher cross-validated mean, the S-learner | The simplest rule, and the interpretable model, readable by native TreeSHAP | The gap sits inside the noise, so the mean is not a lead |
Stability first, then interpretability, the forest | Rewards the model that survives a new sample | Loses the easier model to explain |
Final Decision
Why this option won: the rule written before the numbers breaks ties on the tighter band first, then on interpretability. On the seed the rule ran on, the forest's standard deviation is 0.0067 against 0.0110, and the hold-out confirms the pick: the forest held +0.0936 while the S-learner fell to +0.0806, a drop of 0.0121 that only the X-learner's 0.0211 exceeds.
Which trade-offs were accepted: the band-width reason did not survive more draws. Two standard deviations from five folds are themselves noisy, and on twenty folds across four seeds the forest's band widens to ± 0.0081 while the S-learner leads it on 14 of 20, so the 200,000-row hold-out is what makes the pick credible after the fact, and the band is at most corroboration. The S-learner is recorded as the interpretable near-tie and the stronger cross-validated model; serving kept the forest for consistency with the recorded verdict and its hold-out, and the next re-tag should revisit that.
Which assumptions were made: the tie rule of Decision 3.
Engineering Principle
When two results are a statistical tie, the higher mean is not a winner; break the tie on the property that survives a new sample.
Among the uplift models the forest ships because its hold-out held; its tighter band was the rule's reason and did not survive more draws.
Decision 6: Price the policy as a range
Context
A ranking is half a decision. The other half is money, and money depends on two inputs no dataset carries.
Alternatives Considered
Option | Advantages | Disadvantages |
|---|---|---|
One dollar figure | Quotable | Fabricates precision the inputs do not have |
Incremental conversions only | Fully measured | Stops short of the decision a budget owner needs |
Measured conversions plus a swept value band | Separates the measured from the assumed | Two numbers to explain instead of one |
Final Decision
Why this option won: incremental conversions come off the conversion Qini curve with no assumption. Value is swept over $50 to $150 per conversion, anchored on the $116.36 mean spend among Hillstrom's 578 converters, against $0.05 to $0.50 per contact, and every figure ships beside those inputs.
Which trade-offs were accepted: the contact cost is notional, chosen so a budget bites; real email is near free, which would make treat-everyone win by default. The sign of the value band follows from that choice: at near-free contact the policy is positive throughout and the ranking is moot, which is the point, since ranking only matters when a contact costs something, as an impression does on Criteo.
Which assumptions were made: the two ranges themselves, kept as documented constants rather than buried in a function.
Engineering Principle
Separate the measured from the assumed, and let the assumption travel as a range.
The conversions are measured, the dollars are a swept band, and the assumptions print beside every figure.
Implementation Highlights
The decisions above set the rules; this is where each is enforced, and where the pipeline was checked against itself.
Go deeper: the audited harness, bundle contract and two live bugs
The ruler audited first: Qini and AUUC are a from-scratch implementation asserted equal to scikit-uplift's to machine precision, and the 141 tests across 16 files lead with the evaluation math and the data transforms, because a metric bug corrupts every decision at once where a model bug corrupts one score. They run on every push in continuous integration.
Per-fold scores kept: the leaderboard CSV carries each model's five fold-level Qini values beside the mean and standard deviation, so two models can be compared on the same five draws rather than through two summary bands.
Randomisation checked, not trusted: a standardised mean difference per covariate, categoricals one level at a time, 18 checks on Hillstrom and 12 on Criteo; the worst is 0.0153 on Hillstrom and 0.0518 on Criteo against a flag at 0.1, so nothing flagged.
Stratified where it matters: folds and the hold-out stratify on treatment by outcome; the Criteo subsample stratifies on treatment by conversion so the rare positives survive the draw.
Baselines reproduced to the digit: the two baselines were re-scored in each modelling phase and matched their earlier numbers exactly, a machinery check that the folds and the metric had not moved.
Dependencies staged: the three modelling phases added nothing; serving added fastapi, uvicorn and httpx; the demo added streamlit; shap was never added.
The bundle fits on dev and prices on hold-out: the shipped model is fitted on the development split and the value band is projected on the untouched 20 percent, so the reference the dollars are read off was never trained on. The sanity check reproduces the harness to the digit: a $100 Hillstrom budget prices at $836, band −$98 to $1,157.
One policy input at a time: /policy takes exactly one of budget, count or fraction, with optional value and cost overrides, and the value band always sweeps the documented ranges. The Hillstrom score is labelled response_probability with honest_negative set true, so the product does not pretend uplift where uplift lost.
Two bugs found in the browser: matplotlib's GUI backend segfaulted, exit 139, on Streamlit's worker thread under rapid slider reruns, fixed by forcing Agg before pyplot loads; a dollar caption rendered as LaTeX until the $ was escaped.
The bundle rides in the repository: 6.2 MB of public data, committed as a documented gitignore exception, because Criteo's source download is dead and the cloud cannot rebuild it. Each bundle is stamped with the uplift, lightgbm and numpy versions that built it, and loading under different versions prints a warning naming both, so a dependency bump cannot move a score silently.
Wall time: the full selection board runs in about 88 seconds on Hillstrom and 198 on Criteo, the operating-point read in 4 and 14; persisting the bundles takes about 1.5 and 13 seconds, as measured when the serving layer was built.
The baselines reproduced to the digit across phases, every per-fold score is kept, and the priced reference was never trained on.
Challenges and Debugging
Challenge: the hold-out contradicted the cross-validation
What failed: on both datasets the single hold-out reversed the cross-validated order. On Hillstrom the uplift forest posted +0.0661 against the response model's +0.0187; on Criteo the response model posted +0.0973, above the forest's +0.0936 and the S-learner's +0.0806.
Lesson: when a single draw disagrees with a cross-validated band, the band wins; the disagreement is the reason the band exists.
Challenge: the planned explainer would not build, and would not fit
What failed: SHAP was planned. Its numba dependency cannot build against numpy 2.5.1 under Python 3.12, and the chosen model is a NumPy forest that no tree explainer reads.
Lesson: choose the explainer that fits the model chosen, not the one with the familiar name.
Go deeper: how each was traced and what was done
Hold-out, investigation: instead of asking which split is right, the swings were compared with the fold-to-fold spread. On Hillstrom every uplift model's standard deviation already exceeded its mean, so a 12,800-row draw of that metric was expected to swing either way.
Hold-out, root cause: not a bug. Qini is high-variance, and one split is one sample from a wide distribution.
Hold-out, resolution: the band was the arbiter by design: negative on Hillstrom, a slim and split-sensitive positive on Criteo, and the disagreement recorded as evidence for banding rather than a result to reconcile away.
Explainer, investigation: LightGBM computes exact TreeSHAP itself through pred_contrib, the algorithm shap.TreeExplainer delegates to, so a LightGBM winner needs no package; for the S-learner that is the T equals 1 minus T equals 0 contribution difference with the treatment column dropped. None of it reads the forest.
Explainer, root cause: the explainer was chosen before the winner was.
Explainer, resolution: permutation importance: shuffle a feature five times, measure the mean Qini drop and its spread. f0 costs 0.0049 ± 0.0050, f3 0.0022 ± 0.0037; f1 is exactly unused; f5 and f11 are slightly negative, f11 inside its own noise and f5 just outside it. f0 leads on the mean and the ordering below it is inside the bands. The features are anonymised, so the reading is which signal, not what it means.
Neither challenge was a bug; both were the metric and the model behaving as their variance and their construction predict.
Results and Evaluation
Every number here, and every tile in Figure 2, comes from one configuration. The board is five-fold stratified cross-validation at seed 42 on the development split, 51,200 rows on Hillstrom and 800,000 on Criteo, with normalised Qini on the primary outcome visit as mean and standard deviation across folds. The hold-out, 12,800 and 200,000 rows, is scored once. AUUC tracks Qini throughout and sits in the deep-dive.
Does uplift beat the classifier on 64,000 customers? No
Model | Qini, five-fold CV | Qini, hold-out |
|---|---|---|
treat_everyone | +0.0000 ± 0.0000 | +0.0000 |
response_model, shipped | +0.0197 ± 0.0170 | +0.0187 |
s_learner | +0.0088 ± 0.0174 | +0.0410 |
t_learner | +0.0134 ± 0.0214 | +0.0282 |
x_learner | +0.0085 ± 0.0168 | +0.0505 |
class_transformation | +0.0061 ± 0.0167 | +0.0469 |
uplift_tree | +0.0017 ± 0.0221 | +0.0417 |
uplift_forest | +0.0129 ± 0.0155 | +0.0661 |
No uplift model clears the bar on the mean, and every one has a standard deviation larger than its mean (Figure 4); the baseline's own band overlaps all six, so nothing here is distinguishable from anything, and the rule defaults to the response model rather than crowning it.
The hold-out column flips the order, and at the operating point the disagreement is sharper than the whole-curve numbers suggest: in the top 10 percent of the 12,800-row draw the forest's decile carries +0.151 visit uplift against the baseline's +0.074, with standard errors near 0.02. One draw does not overturn five, and the rule was written before any of them, so the chosen model is the response model, labelled a negative: on this much data, do not deploy an uplift model, and do not read the hold-out as proof that one would work. The disagreement is why Limitations asks for more draws.
On a million rows? A tie
Model | Qini, five-fold CV | Qini, hold-out |
|---|---|---|
treat_everyone | +0.0000 ± 0.0000 | +0.0000 |
response_model, the baseline | +0.0877 ± 0.0074 | +0.0973 |
s_learner | +0.0927 ± 0.0110 | +0.0806 |
t_learner | +0.0703 ± 0.0168 | +0.0626 |
x_learner | +0.0729 ± 0.0145 | +0.0518 |
class_transformation | +0.0776 ± 0.0049 | +0.0810 |
uplift_tree | +0.0784 ± 0.0055 | +0.0876 |
uplift_forest, shipped | +0.0905 ± 0.0067 | +0.0936 |
Two of six uplift models clear the bar on the mean, by 0.0050 and 0.0028, and both bands overlap it (Figure 5). T and X, the two learners that fit a separate model on the 150,000-row control arm, land under the bar, while the S-learner pools both arms and the forest splits on uplift directly; the base-learner swap in the deep-dive supports that reading indirectly. On the single hold-out draw the baseline scores highest at +0.0973, so the edge cross-validation finds does not reproduce out of sample on the whole curve. The honest reading is a tie, with the forest its most stable member on the seed the rule was applied to; where a budget would break the tie is the operating point, below.
On all fourteen million? Narrowly, yes
The subsample is a choice, so the three models the verdict turns on were re-run on the whole Criteo file under the same protocol: 11,183,673 development rows in five stratified folds, a 2,795,919-row hold-out scored once, balance still passing with a worst standardised mean difference of 0.049.
Model | Qini, five-fold CV, all rows | Qini, hold-out | Folds above the bar |
|---|---|---|---|
response_model, the baseline | +0.0846 ± 0.0013 | +0.0844 | not applicable |
s_learner | +0.0899 ± 0.0021 | +0.0893 | 5 of 5 |
uplift_forest, shipped | +0.0881 ± 0.0016 | +0.0887 | 5 of 5 |
Fourteen times the rows shrink the bands about five-fold, and the picture resolves. Both uplift models clear the baseline on every fold, by 0.0035 for the forest and 0.0053 for the S-learner, p below 0.001 for each on a paired test, and by about the same on the hold-out. The edge is small, four to six percent of the baseline's Qini. It is no longer inside the noise. The S-learner against the forest stays a tie (gap 0.0018, p 0.07), and the rule re-applied returns the forest on the tighter band. The full field, the bundle and the demo remain on the one-million-row subsample; the scale check is what turns "draws level" into "pulls clear, narrowly".
The ordering of the designs is the base learner's, not the data's: re-run over a linear base on the same folds, the T-learner, last under LightGBM, is the best design on Criteo and the S-learner falls to the bar, while the small edge over the response model survives both learners. The full swap is in the deep-dive.
Does the top of the ranking respond more?
Figure 6 matters more than the aggregate Qini, because the policy spends from the top of the list. The profile is not perfectly monotone; the small return at decile ten is the noise of estimating a one-point effect on a 4.72 percent base rate. The response model's top decile sits inside the same whisker, which is why the operating point, not this chart, decides anything.
Where the budget spends, which ranking wins?
The coefficient scores the whole curve; a budget acts on its head. Figure 7 reads the two hold-out curves there: through the first tenth they run together, the forest ahead by 159 visits at the cut, and the baseline's higher coefficient is earned past 40 percent, where no budget goes.
A committed script reads both rankings at that operating point on the same 200,000-row draw:
Ranking, top 10 percent of the Criteo hold-out | Top-decile visit uplift | Incremental visits, share of treat-everyone | Incremental conversions | Net value, point | Net value, band |
|---|---|---|---|---|---|
uplift forest, shipped | +0.072 ± 0.008 | 1,240, 68 percent | +23.8 ± 52 | $767 | −$8,811 to $2,567 |
response model, the baseline | +0.063 ± 0.010 | 1,081, 59 percent | +44.2 ± 54 | $3,147 | −$7,788 to $5,636 |
The ± here is a standard error on one draw, not a fold band. On visits, the outcome the forest was fitted on, it captures more at the operating point: 1,240 against 1,081 incremental visits, 68 against 59 percent of what treating everyone captures, and a top decile of 7.2 against 6.3 points, a lead inside one standard error. On the priced outcome the point favours the baseline and neither number carries much: +23.8 is 389 treated conversions minus 57 control conversions scaled by the 6.4 to 1 arm ratio inside the top 20,000, so one control conversion moves it by about six, and both standard errors exceed their estimates.
So the forest ships for two reasons that survive the tie, and neither is that it beat the baseline. At the operating point it is level or ahead on visits, the outcome it was fitted on. And its score has the units of the decision: an uplift can be thresholded at zero, and the serving checks show the near-zero and negative scores as the customers the forest would not contact, which a response probability cannot express. On this data the response model would spend the same 10 percent nearly as well, and the page says so.
What drives the ranking
Figure 8 names the forest's drivers by permutation: shuffle one feature five times and read the mean drop in hold-out Qini with its spread.
What is the policy worth?
Figure 9 is that policy live: targeting the top 10 percent of the Criteo hold-out by predicted uplift spends $2,000 at $0.10 a contact on 20,000 customers for +23.8 incremental conversions, $767 at $116.36 per conversion, inside the value band of the table above. The value band crosses zero for two reasons a budget owner should see side by side: the contact cost is notional and chosen so the budget bites, and the conversion count behind the point is thinner than its own sampling error. That range, not one reassuring figure, is the message.
Go deeper: AUUC, the base-learner swap, drivers and serving checks
Dataset | Model | AUUC, five-fold CV | AUUC, hold-out |
|---|---|---|---|
Hillstrom | response_model | +0.0111 ± 0.0099 | +0.0112 |
Hillstrom | uplift_forest | +0.0072 ± 0.0090 | +0.0395 |
Criteo | response_model | +0.0345 ± 0.0030 | +0.0383 |
Criteo | s_learner | +0.0364 ± 0.0044 | +0.0316 |
Criteo | uplift_forest | +0.0355 ± 0.0027 | +0.0367 |
Feature | Qini drop when shuffled, mean ± std over five shuffles |
|---|---|
f0 | 0.0049 ± 0.0050 |
f3 | 0.0022 ± 0.0037 |
f6 | 0.0019 ± 0.0042 |
f10 | 0.0017 ± 0.0012 |
f8 | 0.0016 ± 0.0018 |
f9, f4, f2 | 0.0008 ± 0.0013, 0.0007 ± 0.0005, 0.0005 ± 0.0011 |
f7 | 0.0001 ± 0.0003 |
f1 | 0.0000 ± 0.0000, unused |
f11, f5 | −0.0007 ± 0.0009, −0.0009 ± 0.0006 |
The base-learner swap. Every meta-learner and the response model share one LightGBM parameter set, so the same four designs were re-run over a deliberately simple base, a linear model over one-hot categoricals and scaled numerics, on the same folds. On Criteo the T-learner, the worst design under LightGBM at +0.0703 ± 0.0168, is the best under the linear base at +0.0960 ± 0.0123 against a linear response bar of +0.0840 ± 0.0074, and the S-learner falls to that bar. On Hillstrom nothing separates under either base. The ranking of designs is a fact about the learner; what survives both learners is that uplift's edge over the response model is small on a million rows. The swap also says something about why T and X lose under LightGBM: a twelve-parameter linear model fitted on the 150,000-row control arm does not overfit it, and a 200-tree booster can.
The two baselines reproduced their earlier phase numbers to the digit on the final board, and the whole board reproduced again when it was re-run to persist the per-fold values. Read fold by fold on Criteo, the forest beats the response model on 3 of 5 folds (by −0.0005, −0.0056, +0.0136, +0.0056 and +0.0010), the S-learner beats it on 4 of 5, and the S-learner beats the forest on 4 of 5; on Hillstrom the forest beats the response model on 2 of 5 and the S-learner on 1 of 5. Read as paired t-tests on those five folds, no pair separates: forest against response p 0.43 on Criteo and 0.38 on Hillstrom, S-learner against response 0.37 and 0.40, S-learner against forest 0.36 and 0.51. Re-splitting the development rows under three more seeds and pooling with seed 42 gives twenty folds per model: on Criteo the response model sits at +0.0876 ± 0.0055, the S-learner at +0.0922 ± 0.0099 and the forest at +0.0889 ± 0.0081, the forest ahead of the bar on 13 of 20 folds and the S-learner ahead of the forest on 14 of 20; on Hillstrom the response model sits at +0.0194 ± 0.0166, the forest at +0.0118 ± 0.0126 and the S-learner at +0.0098 ± 0.0141, the baseline ahead of the forest on 14 of 20. The rule re-applied to the pooled folds returns the same two verdicts, by a narrower margin. The paired read says what the bands say: nothing on Hillstrom, a slim and unsteady lead on Criteo, and a tie-break on band width that the hold-out, not the folds, makes credible. Serving was verified live: /health lists both models; on 2,000 Criteo hold-out rows /score returned 847 distinct values in the range −0.004 to 0.147, the near-zero and slightly negative scores being the customers the forest would not contact; /policy at a $100 budget priced Criteo at +19.2 incremental conversions and $2,138 with a band of $462 to $2,835, and Hillstrom at $836 with a band of −$98 to $1,157, carrying the honest-negative note. The hosted demo reproduced the phase 8 hold-out Qini on both datasets, +0.0936 on Criteo and +0.0187 on Hillstrom.
The fold band overlaps the baseline at a million rows, the simpler model wins at 64,000, and the forest ships on the operating point.
Business Impact
Never deployed, on anonymised public experiments, so no return is booked; the product is a decision framework and a measured boundary on when to pay for it.
The decision it enables: rank by incremental effect and cut at a budget, so spend goes to the customers a contact moves rather than the ones a response model rewards for converting anyway; on Criteo the forest's top decile carries about seven points of observed uplift against a 1.08-point average, with the response model's top decile inside the same whisker.
What the money looks like: a value band under stated inputs, never a figure: the top 10 percent of the Criteo hold-out prices at $767 inside −$8,811 to $2,567, with a sampling error on the conversions larger than the point, and the response model prices higher on the same draw inside the same noise.
What the negative is worth: on 64,000 customers nothing is distinguishable from the simpler model, so the right call is to skip the uplift model and target likely responders until more data says otherwise; knowing that before the build is the cheapest outcome such a project can have.
Where the edge is: on a million rows the forest is level with the baseline inside the fold bands and trails it on the hold-out; on all fourteen million it clears the baseline on every fold, by four to six percent of its Qini. The fundable claim is "level at a million rows, clear narrowly at fourteen", never "beats".
The product is a rule for when a complex model earns deployment: no at 64,000 customers, barely at a million, narrowly yes at fourteen million.
Limitations
An independent build on two public randomised experiments; the table names what that data cannot establish.
Category | What remains unsolved | Why | What would be required |
|---|---|---|---|
Data | Two datasets that differ in size, domain, base rate and effect size at once, and no clear sleeping-dog signal | Everything varies between the two experiments, so "uplift needs scale" is the leading reading, not a proven cause; only 4 of Criteo's 120 feature-decile cells dip below zero | More experiments that vary one thing at a time, and data where a treatment measurably harms some customers |
Evaluation | No per-row label, and a metric that disagreed with itself across splits | Every number is aggregate; on both datasets the single hold-out reversed the cross-validated order, and twenty folds across four seeds did not separate any pair | A larger hold-out, or a second hold-out drawn under another seed, since more folds did not settle it |
Selection | A tie rule that is a heuristic, and a tie-break that did not survive more draws | The tie rule is a one-standard-deviation heuristic, and on twenty folds the forest's band is no longer the tighter one, so the Criteo pick rests on one hold-out draw and a verdict recorded first | Re-apply the rule at the next re-tag with the S-learner, the stronger cross-validated and the interpretable model, as the candidate |
Modelling | The ranking of designs depends on the base learner | Under LightGBM the S-learner leads and the T-learner is last; under a linear base the order reverses; only the size of the edge over the baseline survives both | A board that crosses designs with base learners, and a tuned linear base rather than a default one |
Deployment | Never deployed; the projection is offline | Acting on the policy changes who is targeted, so realised value is not measured by this harness; the bundles are stamped and warn on a version mismatch, but they remain pickles tied to those versions | The online loop, which is the next system rather than this one |
Scale | The full field, the bundle and the demo sit on a one-million-row subsample of 13,979,592 | The subsample kept each phase tractable, but the edge resolved only on all rows, so the shipped bundle is fitted on the split where the edge is still inside the noise | The full field, the bundle and the demo refitted on all rows |
Assumptions | Value per conversion and cost per contact | $116.36 rests on 578 right-skewed orders with a median of $80.79; the contact cost is notional so the budget bites | Real cost and value inputs from the business |
Explainability | Which feature, not what it means | Criteo's twelve features are anonymised floats, and the forest is read by permutation importance because no tree explainer parses it | Named features, or the S-learner near-tie with LightGBM's native TreeSHAP |
Every number holds for these two experiments and these exact treatments; the ones a real deployment would change are named here, not hidden.
Lessons Learned
Lesson 1: Measure the decision, not the metric the data can afford
Context: the familiar metric, a per-row error, does not exist here, and the next, response AUC, answers the wrong question.
Engineering principle: choose the metric that mirrors the operational decision, here the ranking the budget follows, even when it is noisier and harder to summarise.
Generalization: a metric chosen out of habit measures what is convenient; one chosen from the decision measures what will be acted on.
Lesson 2: A model is only as good as the cheapest thing it beats
Context: making the naive baselines first-class competitors turned "uplift did not help on 64,000 customers" from a footnote into the headline.
Engineering principle: build the trivial baseline first and make it the thing to defeat; if the complex model cannot, that is the finding.
Generalization: the credibility of a positive result is set by how visibly the negative one would have been reported.
Lesson 3: On a noisy metric, the band is the result
Context: every disagreement between the folds and the hold-out resolved the same way, and the S-learner and forest tie was broken on the hold-out and the band, not on a decimal inside it. A standard error turned the dollar point into a number smaller than its own noise, and fourteen times the rows shrank the band five-fold, which is what let the Criteo edge be read at all.
Engineering principle: report the uncertainty as the primary quantity, score the hold-out once, and put a standard error on any number a single draw produces.
Generalization: a point estimate on a high-variance metric is a headline waiting to be wrong.
Lesson 4: Score the curve where the budget spends
Context: the response model's higher hold-out coefficient on Criteo was earned past 40 percent of the population, where no budget goes; through the first tenth the two rankings coincide, and on the fitted outcome the forest is ahead there.
Engineering principle: when a policy acts on the head of a ranking, evaluate the head; a whole-range summary can rank models by the part of the range nobody uses.
Generalization: the metric that decides deployment is the one read at the operating point, and it can disagree with the metric that decides the leaderboard.
Future Improvements
Go deeper: the five roadmap extensions, and the stack
Observational causal inference: propensity weighting and backdoor adjustment, to lift the randomised-data restriction.
Multi-arm and continuous treatments: including the R-learner left optional in v1, to move past one binary arm.
The full field on all rows: the scale check covers three models on the full Criteo file; the other five, the bundle and the demo still sit on the subsample.
Uplift on the conversion outcome: the rarer signal, where the value machinery matters more and the bands will be wider.
Designs crossed with base learners: the linear swap showed the ordering of designs is the learner's; a tuned linear base and the direct models under a second learner would show how much. The T/X control-arm reading, supported indirectly by the swap, still wants a direct test.
Technology Stack
Category | Technologies |
|---|---|
Machine learning | LightGBM base learner; hand-rolled S, T and X meta-learners, uplift tree and forest, and class transformation over NumPy; scikit-uplift for loaders and reference metrics |
Evaluation | From-scratch Qini, AUUC, decile and incremental-value harness; permutation importance |
Backend | FastAPI and Uvicorn, /score, /policy, /health; stdlib pickle bundles |
Demo | Streamlit and matplotlib, in process over the library |
Environment | Python 3.12, uv with a lockfile, Ruff, pytest |
Deployment | Streamlit Community Cloud, free tier |
References
Resource | Link |
|---|---|
GitHub | |
Live demo | |
Documentation | |
Datasets | Hillstrom MineThatData Email Challenge; Criteo Uplift Prediction; both under their own terms, Hillstrom loaded through scikit-uplift |