ROC vs Precision-Recall Curve: When Accuracy Lies to You

Same model. Same test set. Two curves that tell almost opposite stories.

A few years ago, while building a fraud detection model for insurance claims, I hit a wall that every data scientist eventually hits. My model came back with 99%+ accuracy. Leadership loved the number. I did not trust it for a second.

Here is why. Fraud in insurance claims is rare, often less than 2% of all claims. A model that predicts "not fraud" for every single claim would already score above 98% accuracy, and it would catch exactly zero fraud. Accuracy was lying to me, and it took a shift to a different pair of curves to see the truth.

The problem with the confusion matrix's favorite child

Accuracy answers one question: out of everything, how much did the model get right? That question makes sense when your classes are balanced, roughly 50/50. It falls apart when they are not.

In imbalanced problems like fraud detection, churn prediction, or rare disease diagnosis, the minority class is the one you actually care about. A model can ignore it almost entirely and still look brilliant on paper. This is the trap I built my fraud detection portfolio project around, recreating real-world insurance fraud detection methodology on synthetic data with logistic regression and XGBoost, specifically to show how the evaluation choice changes the story.

So we reach for something better. Two candidates show up in every textbook: the ROC curve and the Precision-Recall curve — and the ROC vs Precision-Recall curve question is exactly where most people get imbalanced classification wrong. They look similar. They are not interchangeable, and mixing them up is one of the most common mistakes I see in interview take-homes and production dashboards alike.

Setting up an honest test

To make this concrete instead of theoretical, I ran an actual experiment. I generated a synthetic dataset of 6,000 samples designed to mimic a fraud-detection scenario, where only 1.9% of cases are the positive class (fraud). I trained a simple logistic regression on it and evaluated the held-out test set two ways: once with a ROC curve, once with a Precision-Recall curve.

The results:

• ROC-AUC: 0.82 — a solidly "good" score by most rules of thumb

• PR-AUC (Average Precision): 0.35 — a much less flattering number, for the exact same model

  
  

Same model. Same predictions. Same test set. One curve says "pretty good." The other says "be careful." Both are correct. They are just answering different questions.

ROC: how well do you separate the classes?

The ROC curve stays close to the top-left corner even when false positives pile up, because its x-axis is diluted by a huge pool of true negatives.

The ROC curve plots True Positive Rate against False Positive Rate at every possible decision threshold.

• True Positive Rate is recall. Out of all actual positives, how many did you catch.

• False Positive Rate is out of all actual negatives, how many you wrongly flagged.

The area under this curve, AUC-ROC, has a clean interpretation: it is the probability that the model ranks a randomly chosen positive case above a randomly chosen negative case. An AUC of 0.5 means your model is guessing. An AUC of 1.0 means perfect separation.

Here is the catch. False Positive Rate has all negatives in its denominator. When negatives vastly outnumber positives, as they do at a 98-to-2 ratio, that denominator is huge. So even a large number of false positives barely moves the rate. In the chart above, a false positive rate of just 10% still translates into hundreds of false alarms in absolute terms, because 10% of a very large negative class is still a lot of cases. The ROC curve can look excellent while your model quietly drowns a fraud investigation team in false alarms.

This is not a flaw in the math. It is a mismatch between what the curve measures and what you actually care about.

PR: how much can you trust a positive prediction?

The PR curve has no true negatives to hide behind, so it exposes exactly where precision collapses as you chase more recall.

The Precision-Recall curve plots Precision against Recall.

• Precision is out of everything you flagged as positive, how many actually were.

• Recall is the same as before: out of all actual positives, how many you caught.

Notice what is missing from that pair: true negatives, anywhere. PR curves do not care how well you handle the majority class. They only care about the class you are chasing. That is exactly what you want when the majority class is huge and uninteresting, and the minority class is the one with consequences.

In the experiment above, this is exactly where the honest story shows up. The PR curve starts high (when the model only flags its most confident predictions, precision is decent), but as you push recall higher to catch more of the actual fraud, precision falls off sharply. That drop is the real tradeoff a fraud investigation team lives with every day: catch more fraud, but drown in more false alarms per fraud case caught.

ROC vs Precision-Recall Curve: Which One Should You Use?

A simple rule of thumb, the kind I wish someone had told me directly instead of implying:

Use ROC when classes are roughly balanced, or when you genuinely care about both classes equally. Medical screening, where both false positives and false negatives carry real, comparable cost, is a reasonable ROC scenario.

Use PR when the positive class is rare and is the class you actually care about. Fraud detection, spam filtering, churn prediction, rare disease detection, intrusion detection. Anywhere the "interesting" outcome is a small slice of the data, and flooding people with false positives has a real cost.

A useful gut check: if your model's ROC-AUC looks great but you feel uneasy about it, plot the PR curve. If it drops off a cliff, like it does above, your instinct was right. The ROC curve was flattering you.

The detail almost everyone skips: the baseline moves

One subtlety that separates a solid understanding from a shaky one. A random classifier gives you a diagonal line on ROC, always, regardless of class balance, because both axes are rates computed independently within each class. That is not true for PR.

The random baseline for a PR curve equals the proportion of positives in your dataset. In our experiment, with 1.9% positive cases, a classifier that guesses at random would sit at roughly 0.019 precision across the board, not 0.5. Our model's PR-AUC of 0.35 needs to be read against that 0.019 baseline, not against some fixed universal benchmark.

This has a very practical consequence: you cannot compare PR-AUC scores across two different datasets, or even across two different train-test splits, unless you also report the base rate. I learned this the hard way comparing PR-AUC across quarterly fraud data where the underlying fraud rate had shifted between quarters. The model had not gotten worse. The problem had gotten harder. If you only report the number, you will draw the wrong conclusion.

What I actually do now

When I build an imbalanced classifier, I plot both curves, every time, and I report the base rate next to the PR-AUC so the number means something to whoever reads it later. ROC tells me how well the model separates signal from noise in principle. PR tells me what happens when that model meets the real world, where the vast majority of what it sees is noise, and every false alarm has a human on the other end of it.

Neither curve is the "right" one. They answer different questions. The mistake is picking the curve that makes the model look good instead of the curve that answers the question you are actually being asked.

Accuracy told me my fraud model was excellent. The PR curve told me exactly where it would break under pressure. I trust the second story more, and now I always ask for it first.

Code and full experiment for this post are adapted from my fraud detection portfolio project, where I recreate real-world imbalanced classification methodology using synthetic data.

View on Github