I've made a binary classification model using LightGBM. The dataset was fairly imbalnced but I'm happy enough with the output of it but am unsure how to properly calibrate the output probabilities. The baseline score of the model from sklearn.dummy.DummyClassifier is:

dummy = DummyClassifier(random_state=54)

dummy.fit(x_train, y_train)

dummy_pred = dummy.predict(x_test)

dummy_prob = dummy.predict_proba(x_test)
dummy_prob = dummy_prob[:,1]

print(classification_report(y_test, dummy_pred))

              precision    recall  f1-score   support

           0       0.98      0.98      0.98    132274
           1       0.02      0.02      0.02      2686

   micro avg       0.96      0.96      0.96    134960
   macro avg       0.50      0.50      0.50    134960
weighted avg       0.96      0.96      0.96    134960

The output of the model is below and am ok with the results:

print(classification_report(y_test, y_pred))

              precision    recall  f1-score   support

           0       1.00      0.95      0.97    132274
           1       0.27      0.96      0.42      2686

   micro avg       0.95      0.95      0.95    134960
   macro avg       0.63      0.95      0.70    134960
weighted avg       0.98      0.95      0.96    134960

I want to use the output probabilities so I thought I should look at how well the model is calibrated as tree based models can often be not calibrated very well. I used sklearn.calibration.calibration_curve to plot the curve:

import matplotlib.pyplot as plt
from sklearn.calibration import calibration_curve

gb_y, gb_x = calibration_curve(y_test, rf_probs, n_bins=10)

plt.plot([0, 1], [0, 1], linestyle='--')
# plot model reliability
plt.plot(gb_x, gb_y, marker='.')

Calibration curve from model output

I Tried Platt scaling to the data, i.e. fitting a logistic to the validation set output probabilities and apply it to the test data. While it is more calibrated, the probabilities are restricted to a max of approx 0.4. I would like the output to have a good range, i.e. people with low and high probabilities.

Calibration curve after calibration

Does anybody know about how I would go about this?


I would suggest not changing the (calibrated) predicted probabilities. Some further points:

  1. While calibrated probabilities appearing "low" might be counter-intuitive, it might also be more realistic given the nature of the problem. Especially when operating in an imbalanced setting, predicting that a particular user/person has a very high absolute probability of being in the very rare positive class might be misleading/over-confident.
  2. I am not 100% clear from your post how the calibration was done. Assuming we did repeated-CV $2$ times $5$-fold cross-validation: Within each of the 10 executions should use a separate say $K$-fold internal cross-validation with ($K-1$) folds for learning the model and $1$ for fitting the calibration map. Then $K$ calibrated classifiers are generated within each execution and the outputs of them are averaged to provide predictions on the test fold. (Platt's original paper Probabilities for SV Machines uses $K=3$ throughout but that is not a hard rule.)
  3. Given we are calibrating the probabilities of our classifier it would make sense to use proper scoring rule metrics like Brier score, Continuous Ranked Probability Score (CRPS), Logarithmic score too (the latter assuming we do not have any $0$ or $1$ probabilities being predicted).
  4. After we have decided the threshold $T$ for our probabilistic classifier, we are good to explain what it does. Indeeed, the risk classification might suggest to "treat any person with risk higher than $0.03$"; that is fine if we can relate it to the relevant misclassification costs. Similarly, if misclassification costs are unavailable, if we use a proper scoring rule like Brier, we are still good; we have calibrated probabilistic predictions, anyway.

If you want the output to have a good range, you should definitely tackle the imbalanced data problem. You can choose - depending on your data and especially the number of occurence by class - either to oversample the underrepresented class (be careful this leads to lower the variance of this class). Or you can undersample the overrepresented class (the disadvantage of this method is that you don't use all your data and you may miss important samples).

A third method exists: weights your data. Thus you give more weight to the underrepresented class. It allows you to use all your data and avoid to change the variance.

  • 3
    $\begingroup$ Over/under-sampling is not necessarily good advice here. It would require us to recalibrate the probabilities after training such that that it corresponds to reality. It is doable but it must be done. As the classifiers prediction reflect the underlying training distribution if we move from a 1-99 split to a 50-50 split we need to calibrate the probabilities so it is back again a 1-99 split as a 50-50 is unrealistic of the actual deployment environment (i.e. reality). $\endgroup$ – usεr11852 Aug 28 '19 at 17:31
  • $\begingroup$ This is true, we have to be aware of this deviation from reality. I was trying to give a solution to have a [0,1] probability range. $\endgroup$ – Samos Aug 29 '19 at 8:43

Instead of performing a sigmoid/Platt regression, you can try an isotonic one, as described here: https://scikit-learn.org/stable/modules/calibration.html#isotonic I have had better results with isotonic regressions, by which I mean that the calibrated model spans the whole probability range and is closer to a linear relation.

The article that I referenced also describes the CalibratedClassifierCV which you can use to perform the calibration with both sigmoid and isotonic regressors.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.