6
$\begingroup$

I run xgboost and elastic-net on the same dataset for a classification problem, say we have

y_train, y_test
y_train_xgboost_prediction, y_test_xgboost_prediction
y_train_elastic_net_prediction, y_test_elastic_net_prediction

I find xgboost gives prediction with mean a lot higher than actual label, which is

average(y_train_xgboost_prediction) > average(y_train)
average(y_test_xgboost_prediction) > average(y_test)

While elastic-net gives prediction with average similar to average of actual label, which is

average(y_train_elastic_net_prediction) ~= average(y_train)
average(y_test_elastic_net_prediction) ~= average(y_test)

Actually xgboost has AUC higher than elastic-net, which model should we prefer in this case? How do we fix the bias in xgboost prediction?

$\endgroup$
7
  • 3
    $\begingroup$ Good question. Boosting methods usual do not give well calibrated probabilistic predictions (e.g. see Niculescu-Mizil & Caruana). That said, they can be calibrated on a follow up step (e.g. through isotonic regression). Choice of the model depends on what you want to do with it/use its estimates. Do we care more about the order or the marginal probabilities of our estimates? (Probably I should write this as an answer at some point.) $\endgroup$ – usεr11852 Sep 7 '19 at 1:57
  • $\begingroup$ @usεr11852 - You should post that as an answer! $\endgroup$ – jbowman Sep 7 '19 at 3:00
  • $\begingroup$ @jbowman: Done. $\endgroup$ – usεr11852 Sep 7 '19 at 12:15
  • $\begingroup$ What objective did you use? $\endgroup$ – Michael M Sep 7 '19 at 13:02
  • $\begingroup$ I used AUC at first. But I also want to keep the original scale of probability prediction. So I'm thinking to do some post-processing based on xgboost prediction. $\endgroup$ – Salty Gold Fish Sep 7 '19 at 15:45
5
$\begingroup$

What is described is not very surprising. Boosting methods usually do not give well calibrated probabilistic predictions (e.g. see Caruana et al. (2004) "Ensemble Selection from Libraries of Models", Niculescu-Mizil & Caruana (2005) "Predicting good probabilities with supervised learning"). GLM-based methods provide more consistent marginal probabilities as they have a direct probabilistic connection.

That said, probabilistic estimates can be calibrated on a follow up step (e.g. through isotonic regression or beta calibration). Please note that these methods are usually applied on a separate hold-out sample. Check the paper from Kull et al. (2017) "Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers" irrespective of using beta calibration or not, it gives a very nice modern exposition of the matter.

The choice of the model depends on what we want to do with it/use its estimates. Do we care more about the order or the marginal probabilities of our estimates? If, for example, we care about the "fairness" of our prediction, the marginal estimates are more important. On the other hand, if we want to pick the "top X more probable" items for a particular treatment, AUC are more important. Probability calibration techniques try to bridge that gap to certain extent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.