Honest About Uncertainty: Prediction Intervals and Quantile Regression in Plain Language
A model that only gives you an average is telling you half the story. This is a plain-language follow-up to my SOA predictive modeling research on predicting uncertainty, written for anyone who wants their models to say not just "what's likely," but "how sure are we?"
Why an average isn't enough
Most predictive models are trained to estimate an average. That's useful, but it hides the part of the picture that often matters most. Two patients can have the same expected health care cost and very different risks. One is almost certain to land near the average. The other has a real chance of a very large bill.
In actuarial work, that difference is the whole job. Reserves, pricing, and capital planning all care about the tail: the bad year, the large claim, the outcome you hope doesn't happen but have to be ready for. A single number can't carry that. A range can.
What a quantile actually is
A quantile is a cut point. The median, or 50th percentile, is the value with half of the outcomes at or below it. The 90th percentile is the value with 90% of outcomes at or below it. If a model predicts the 90th percentile of next year's cost for a patient, it's saying: "We expect the cost to be at or below this number nine times out of ten."
Quantile regression is simply regression that targets one of those cut points instead of the mean. Run it for the 5th percentile and the 95th percentile, and you have a lower and upper bound. Together they form a prediction interval that's meant to contain about 90% of outcomes.
The trick: a lopsided loss
Ordinary regression penalizes errors symmetrically. Quantile regression uses a lopsided penalty, often called the pinball loss. For a target quantile τ (written as a number between 0 and 1), the loss on one prediction is:
- τ × (actual − predicted) when the actual value is above the prediction, and
- (1 − τ) × (predicted − actual) when the actual value is below it.
Take τ = 0.9. Predicting too low costs 0.9 per unit of error, and predicting too high costs 0.1. Under-predicting is nine times as expensive, so the model learns to aim high, right up to the point where about 90% of outcomes fall below its prediction. That's the whole idea in one sentence: change the shape of the penalty, and you change what the model is aiming at.
Doing it with gradient boosting
In my SOA paper, I paired quantile regression with gradient boosting, an ensemble of small decision trees where each new tree corrects the errors of the ones before it. Popular libraries support this directly. In scikit-learn, it looks like this:
from sklearn.ensemble import GradientBoostingRegressor
lower = GradientBoostingRegressor(loss="quantile", alpha=0.05)
upper = GradientBoostingRegressor(loss="quantile", alpha=0.95)
lower.fit(X_train, y_train)
upper.fit(X_train, y_train)
low, high = lower.predict(X_test), upper.predict(X_test)
coverage = ((y_test >= low) & (y_test <= high)).mean()
print(f"Share of test outcomes inside the interval: {coverage:.1%}")LightGBM and XGBoost offer quantile objectives too. The library matters less than the habit on the last two lines: always measure how often the interval actually contains the truth on data the model hasn't seen.
Three things that trip people up
Coverage is a claim you have to check. A "90% interval" is only a label until you test it. If held-out data lands inside it far less often than 90% of the time, the interval is overconfident, and it's better to know that before anyone relies on it.
Quantiles can cross. When the lower and upper models are trained separately, the predicted 5th percentile can occasionally end up above the 95th for a few rows. It's worth checking for and handling, for example by sorting the two predictions or revisiting the model.
Percentiles don't add up like averages. As I noted in the paper's discussion, you can sum expected costs across a group of patients, but you can't sum their individual 90th percentiles and get the group's 90th percentile. Group-level risk needs its own model or a simulation.
If you need stronger guarantees, look into conformal prediction, a family of methods that wraps around an existing model and calibrates its intervals on held-out data so their coverage holds under fairly mild assumptions.
Uncertainty as a data-quality tool
The second application in my paper turned the same idea around. If a model has learned what a typical range looks like, a record that falls far outside its extreme quantiles is worth a second look. It might be a data-entry mistake or a genuinely unusual case. The right move is to flag it for a person to review, not to quietly delete it. Garbage in, garbage out, but only a human can tell which is which.
The honest part is the human part
I think of prediction intervals as a form of honesty. It takes some humility to hand someone a range and say, "Here's what we know, and here's how much we don't." In a world that's starting to automate predictions, that humility is still a human contribution. A model can produce the numbers. Deciding how much to trust them, and explaining that clearly to the people who have to act on them, is the work.
It's also one of my favorite examples of AI as a lever rather than a replacement. The tools make it fast to fit these models. The judgment about what to measure, how to check it, and what it means is a talent worth growing. If you'd like to practice, try reproducing the notebook from the SOA research page on your own data, and share what you find. That's how we learn from each other.