2
$\begingroup$

I'm running a gradient boosted tree in sklearn and running on some test data. The frequency of positive examples ('1') in the test data should be around 10%, which the prediction is returning correctly. However, if I average the predicted probabilities from predict_proba, I get an average of around 25%.

Shouldn't the predict_proba average be 10% as well? I suppose there is nothing in the model to force this to be true; predict is just making a determination based on whether or not the value is greater than 50%. Is there a way to correct for this?

$\endgroup$

1 Answer 1

1
$\begingroup$

Yes, if model outputs are calibrated, the average of predict_proba should be close to 10%, due to linearity of the mean.

Check out http://scikit-learn.org/stable/modules/calibration.html for some methods to do it.

$\endgroup$
2
  • $\begingroup$ Awesome, exactly what i was looking for. Question - are Multi Layer Perceptrons well calibrated like logistic regression? Would be intuitive that this would be so since the models are so similar. $\endgroup$
    – dashnick
    Jun 3, 2018 at 18:01
  • 1
    $\begingroup$ Brief search reveals that this question is worth a paper :) according to this one dl.acm.org/citation.cfm?doid=1102351.1102430 neural networks are indeed calibrated. However according to this arxiv.org/abs/1706.04599, calibration can be negatively impacted by some of the recent techniques. $\endgroup$
    – psarka
    Jun 4, 2018 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.