A screening classifier reaches 89% accuracy on a held-out test set, and for a batch of cases it labels positive with 57% confidence a reviewer reasonably expects that about 57 of every 100 are genuine. In the run examined below, 73 of every 100 are. The model is accurate, yet its stated probabilities do not match the outcomes they are supposed to describe. This gap between confidence and correctness is miscalibration, and it is a separate property from accuracy: a model can rank cases well while reporting probabilities that are systematically too high or too low.
Calibration asks whether a classifier’s probability outputs mean what they claim: among predictions made with confidence p, the fraction that turn out correct should be close to p. It is measured with reliability diagrams, the expected calibration error (ECE), and proper scoring rules such as the Brier score. A short reproducible experiment on a random-forest classifier shows an ECE of 0.071 that post-hoc scaling reduces to roughly 0.015 while classification accuracy stays essentially unchanged, because calibration adjusts the probabilities without reordering the predictions.
What calibration means
A binary classifier usually outputs a score between 0 and 1 that is treated as the probability of the positive class. The model is calibrated when those scores are honest frequencies: across all cases assigned a confidence near 0.8, close to 80% should belong to the positive class. Accuracy, by contrast, only asks whether the thresholded decision is right. The two come apart often. A model can achieve high accuracy by ranking positives above negatives while attaching probabilities that are uniformly inflated, and a well-ranked model with poor calibration will mislead any downstream step that consumes the probability rather than the label — expected-cost decisions, risk thresholds, or the fusion of several models’ outputs.
Miscalibration has a direction. When the true accuracy in a confidence band is lower than the stated confidence, the model is overconfident; when it is higher, the model is underconfident. Tree ensembles and margin-based classifiers are known to distort probabilities in characteristic ways, which is why the effect is worth measuring rather than assuming.
Measuring miscalibration
The standard visual tool is the reliability diagram. Predictions are sorted into equal-width confidence bins; for each bin the mean predicted confidence is plotted against the empirical accuracy. Perfect calibration lies on the diagonal, where confidence equals accuracy. Points below the diagonal indicate overconfidence, points above it indicate underconfidence, and the vertical distance from the diagonal is the per-bin calibration gap.
A diagram is informative but not a single number. The expected calibration error collapses it to one value: the bin-count-weighted average of the absolute gap between accuracy and confidence across all bins. Lower is better, and zero means the reliability curve lies exactly on the diagonal. ECE depends on the number of bins and on the binning scheme, so it is a comparison tool within a fixed protocol rather than an absolute constant. It also averages away compensating errors, which is why it is best read alongside the diagram and a proper scoring rule.
The Brier score — the mean squared difference between the predicted probability and the 0/1 outcome — is one such proper scoring rule, minimised only when the reported probabilities are both accurate and well calibrated. Log loss (cross-entropy) is another, and it penalises confident mistakes more sharply. Reporting ECE together with Brier score and log loss gives a fuller picture than any one metric alone.
A measured example
To make the numbers concrete, a random-forest classifier of 200 trees (maximum depth 7) was trained on a synthetic dataset of 20,000 examples with 20 features and a 30% positive rate, using a fixed random seed. The data were split into 12,000 training, 4,000 calibration, and 4,000 test examples; the calibration split was reserved for fitting the post-hoc maps described in the next section, and every figure below is computed on the untouched test split. The reliability diagram above is this model’s, and the summary metrics are as follows.
| Method | Accuracy | Brier | Log loss | ECE (15 bins) |
|---|---|---|---|---|
| Uncalibrated | 0.8942 | 0.0843 | 0.2901 | 0.0706 |
| Platt (sigmoid) | 0.8960 | 0.0781 | 0.2640 | 0.0169 |
| Isotonic | 0.8958 | 0.0781 | 0.2674 | 0.0145 |
The uncalibrated ECE of 0.0706 confirms what the diagram shows. The departure from the diagonal runs in both directions: the two lowest-confidence bins are mildly overconfident, while the mid-to-high range from roughly 0.5 to 0.9 is markedly underconfident, with empirical accuracy exceeding stated confidence by 12 to 18 points. This mixed pattern is exactly the kind of structure a single accuracy figure cannot reveal.
Fixing it after training
Calibration can be repaired after training by fitting a small function that maps the model’s raw scores to corrected probabilities, using data the model did not train on. Two classical approaches appear in the table. Platt scaling fits a one-parameter logistic (sigmoid) transform of the score, which works well when the reliability curve has a smooth S-shape but cannot represent more irregular distortions. Isotonic regression fits any non-decreasing step function, so it is more flexible; the trade-off is that it needs more calibration data and can overfit on small sets. For neural networks a further special case, temperature scaling, divides the logits by a single learned constant before the softmax; because a positive constant does not change which class scores highest, the predicted labels — and therefore accuracy — are left untouched.
The measured result is the point worth keeping. Both maps cut ECE to roughly a fifth of its original value — 0.0169 for Platt scaling and 0.0145 for isotonic — and both also improve the Brier score and log loss, since better probabilities lower every proper scoring rule. Meanwhile accuracy barely moves, from 0.8942 to about 0.8960: because a monotonic map preserves the ordering of scores, almost no case crosses the 0.5 decision threshold. Calibration is best understood as a correction to the probabilities, not to the classifier’s ability to separate the classes. The same distinction underlies threshold selection in one-class anomaly detection and the score-versus-decision boundary discussed in the SVM and one-class SVM comparison.
Conclusion
Accuracy and calibration answer different questions. Accuracy asks whether the label is right; calibration asks whether the probability is honest. A model that will feed its probabilities into a cost-sensitive decision, a risk threshold, or an ensemble should be checked with a reliability diagram and summarised with ECE and a proper scoring rule, not accuracy alone. When the probabilities drift, a post-hoc map fitted on held-out data usually restores them at negligible cost to accuracy — a cheap and well-understood step that turns a good ranker into a trustworthy probability estimator. For related evaluation pitfalls with skewed classes, see the discussion of thresholds and metrics in time-series anomaly detection models.
Frequently Asked Questions
Does calibrating a model improve its accuracy?
Not in general. Platt scaling, isotonic regression, and temperature scaling are monotonic transforms of the score, so they preserve the ranking of predictions and therefore leave the thresholded labels — and accuracy — essentially unchanged. In the experiment above accuracy moved only from 0.8942 to about 0.8960. What improves is the quality of the probabilities, reflected in lower ECE, Brier score, and log loss.
When should isotonic regression be preferred over Platt scaling?
Isotonic regression fits an arbitrary non-decreasing function and can correct irregular reliability curves that a single sigmoid cannot, so it tends to win when there is enough held-out calibration data. On small calibration sets its flexibility becomes a liability and it can overfit, in which case the one-parameter Platt sigmoid is the safer choice. Comparing both on a validation split, as in the table above, is the reliable way to decide.
References
- Guo, C., Pleiss, G., Sun, Y., Weinberger, K. Q. “On Calibration of Modern Neural Networks.” ICML 2017. arxiv.org/abs/1706.04599 (accessed 2026-07-28).
- scikit-learn developers. “Probability calibration” (User Guide, version 1.9). scikit-learn.org/stable/modules/calibration.html (accessed 2026-07-28).
- Niculescu-Mizil, A., Caruana, R. “Predicting Good Probabilities With Supervised Learning.” ICML 2005. cs.cornell.edu (accessed 2026-07-28).
- Brier, G. W. “Verification of Forecasts Expressed in Terms of Probability.” Monthly Weather Review, 78(1), 1950. journals.ametsoc.org (accessed 2026-07-28).
Leave a Reply