0
$\begingroup$

I have trained a VGG-16 model toward a binary classification task. The model was trained on equal numbers of abnormal and normal images (n=2000). Literature studies demonstrate that model calibration is performed, when the model is trained on an imbalanced dataset, to rescale the probabilities to reflect the true likelihood of occurrence of the samples of a given class. However, I experimented to see if calibration impacts performance on a model trained on balanced dataset. The model, though, trained on a balanced dataset, underpredicted the positive class as observed from the below figure as the uncalibrated outputs were lying above the y=x diagonal. I observed that on applying various calibration methods, the expected calibration error (ECE) decreased compared to that obtained with the non-calibrated output (Non-calibrated ECE:0.039 vs. Platt calibrated output: 0.02). Also, the calibrated outputs closely followed the y=x diagonal. After calibration, the precision, Kappa, F-score, and MCC metrics increased compared to that obtained by the uncalibrated outputs. enter image description here

I have the following questions:

  1. Since the classes are balanced, the model is not prone to biased learning. How does a calibration error occur in a model trained on a balanced dataset?
  2. Is the calibration error occurring due to the model's capacity toward learning the positive and negative samples?
  3. Is it always necessary to calibrate a model trained on balanced data?
$\endgroup$
1
  • $\begingroup$ I am a bit confused by "when the model is trained on an imbalanced dataset, to rescale the probabilities to reflect the true likelihood of occurrence of the samples of a given class." followed by "However, I experimented to see if calibration impacts performance on a model trained on balanced dataset". Typically you only need scaling if you train on a balanced training set, where the class frequencies in operation are imbalanced, but it is not clear that is the situation being discussed here. $\endgroup$ Jul 28, 2021 at 9:04

1 Answer 1

1
$\begingroup$

Even though the data is balanced, it looks like the model is not trained well enough to accurately identify the positives. So the calibration error is coming from not so perfect model. Try to find the calibration error on a highly accurate model, there might be a very small error. To answer your last question, I think calibration can improve a not-so-perfect model when trained on balanced data. There can be situations where your best model is about say 80% accurate. This might be due to the complicated input data (where models struggle to find good features). In situations like this, based on your experiment, I suppose calibration can further improve your model's prediction accuracy.

$\endgroup$
1
  • $\begingroup$ That makes a lot of sense to me. Thanks!!! $\endgroup$
    – shiva
    Jul 28, 2021 at 10:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.