I've made a binary classification model using LightGBM. The dataset was fairly imbalanced but I'm happy enough with the output of it but am unsure how to properly calibrate the output probabilities. The baseline score of the model from sklearn.dummy.DummyClassifier is:

dummy = DummyClassifier(random_state=54)

dummy.fit(x_train, y_train)

dummy_pred = dummy.predict(x_test)

dummy_prob = dummy.predict_proba(x_test)
dummy_prob = dummy_prob[:,1]

print(classification_report(y_test, dummy_pred))

              precision    recall  f1-score   support

           0       0.98      0.98      0.98    132274
           1       0.02      0.02      0.02      2686

   micro avg       0.96      0.96      0.96    134960
   macro avg       0.50      0.50      0.50    134960
weighted avg       0.96      0.96      0.96    134960

The output summary for the model is below and I am happy with the results:

print(classification_report(y_test, y_pred))

              precision    recall  f1-score   support

           0       1.00      0.95      0.97    132274
           1       0.27      0.96      0.42      2686

   micro avg       0.95      0.95      0.95    134960
   macro avg       0.63      0.95      0.70    134960
weighted avg       0.98      0.95      0.96    134960

I want to use the output probabilities so I thought I should look at how well the model is calibrated as tree-based models can often not be calibrated very well. I used sklearn.calibration.calibration_curve to plot the curve:

import matplotlib.pyplot as plt
from sklearn.calibration import calibration_curve

gb_y, gb_x = calibration_curve(y_test, rf_probs, n_bins=10)

plt.plot([0, 1], [0, 1], linestyle='--')
# plot model reliability
plt.plot(gb_x, gb_y, marker='.')

Calibration curve from model output

I tried Platt scaling to the data, i.e. fitting a logistic to the validation-set output probabilities and applying it to the test data. While it is better calibrated, the probabilities are restricted to a max of approx 0.4. I would like the output to have a good range, i.e. individuals to have low and high predicted probabilities.

Calibration curve after calibration

Does anybody know about how I would go about this?


3 Answers 3


I would suggest not changing the (calibrated) predicted probabilities. Some further points:

  1. While calibrated probabilities appearing "low" might be counter-intuitive, it might also be more realistic given the nature of the problem. Especially when operating in an imbalanced setting, predicting that a particular user/person has a very high absolute probability of being in the very rare positive class might be misleading/over-confident.
  2. I am not 100% clear from your post how the calibration was done. Assuming we did repeated-CV $2$ times $5$-fold cross-validation: Within each of the 10 executions should use a separate say $K$-fold internal cross-validation with ($K-1$) folds for learning the model and $1$ for fitting the calibration map. Then $K$ calibrated classifiers are generated within each execution and the outputs of them are averaged to provide predictions on the test fold. (Platt's original paper Probabilities for SV Machines uses $K=3$ throughout but that is not a hard rule.)
  3. Given we are calibrating the probabilities of our classifier it would make sense to use proper scoring rule metrics like Brier score, Continuous Ranked Probability Score (CRPS), Logarithmic score too (the latter assuming we do not have any $0$ or $1$ probabilities being predicted).
  4. After we have decided the threshold $T$ for our probabilistic classifier, we are good to explain what it does. Indeeed, the risk classification might suggest to "treat any person with risk higher than $0.03$"; that is fine if we can relate it to the relevant misclassification costs. Similarly, if misclassification costs are unavailable, if we use a proper scoring rule like Brier, we are still good; we have calibrated probabilistic predictions, anyway.
  • $\begingroup$ In (3) you say "latter" - do you mean last, or last two? (latter usually applies in the, er, binary case :) ). PS Nice answer! $\endgroup$
    – jtlz2
    Commented Nov 10, 2022 at 19:40
  • 1
    $\begingroup$ Thank you! :) Yeah, I meant the last; cause $\text{log}(0) = -\infty$. $\endgroup$
    – usεr11852
    Commented Nov 10, 2022 at 23:07
  • $\begingroup$ OK cool. I just came across this answer again and came to the same conclusion that it's brilliant - thank you again! $\endgroup$
    – jtlz2
    Commented Nov 29, 2022 at 9:16

Instead of performing a sigmoid/Platt regression, you can try an isotonic one, as described here: https://scikit-learn.org/stable/modules/calibration.html#isotonic I have had better results with isotonic regressions, by which I mean that the calibrated model spans the whole probability range and is closer to a linear relation.

The article that I referenced also describes the CalibratedClassifierCV which you can use to perform the calibration with both sigmoid and isotonic regressors.


If you want the output to have a good range, you should definitely tackle the imbalanced data problem. You can choose - depending on your data and especially the number of occurence by class - either to oversample the underrepresented class (be careful this leads to lower the variance of this class). Or you can undersample the overrepresented class (the disadvantage of this method is that you don't use all your data and you may miss important samples).

A third method exists: weights your data. Thus you give more weight to the underrepresented class. It allows you to use all your data and avoid to change the variance.

  • 4
    $\begingroup$ Over/under-sampling is not necessarily good advice here. It would require us to recalibrate the probabilities after training such that that it corresponds to reality. It is doable but it must be done. As the classifiers prediction reflect the underlying training distribution if we move from a 1-99 split to a 50-50 split we need to calibrate the probabilities so it is back again a 1-99 split as a 50-50 is unrealistic of the actual deployment environment (i.e. reality). $\endgroup$
    – usεr11852
    Commented Aug 28, 2019 at 17:31
  • $\begingroup$ This is true, we have to be aware of this deviation from reality. I was trying to give a solution to have a [0,1] probability range. $\endgroup$
    – Samos
    Commented Aug 29, 2019 at 8:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.