Biased prediction (overestimation) for xgboost

I run xgboost and elastic-net on the same dataset for a classification problem, say we have

y_train, y_test
y_train_xgboost_prediction, y_test_xgboost_prediction
y_train_elastic_net_prediction, y_test_elastic_net_prediction


I find xgboost gives prediction with mean a lot higher than actual label, which is

average(y_train_xgboost_prediction) > average(y_train)
average(y_test_xgboost_prediction) > average(y_test)


While elastic-net gives prediction with average similar to average of actual label, which is

average(y_train_elastic_net_prediction) ~= average(y_train)
average(y_test_elastic_net_prediction) ~= average(y_test)


Actually xgboost has AUC higher than elastic-net, which model should we prefer in this case? How do we fix the bias in xgboost prediction?

• Good question. Boosting methods usual do not give well calibrated probabilistic predictions (e.g. see Niculescu-Mizil & Caruana). That said, they can be calibrated on a follow up step (e.g. through isotonic regression). Choice of the model depends on what you want to do with it/use its estimates. Do we care more about the order or the marginal probabilities of our estimates? (Probably I should write this as an answer at some point.) – usεr11852 Sep 7 '19 at 1:57
• @usεr11852 - You should post that as an answer! – jbowman Sep 7 '19 at 3:00
• @jbowman: Done. – usεr11852 Sep 7 '19 at 12:15
• What objective did you use? – Michael M Sep 7 '19 at 13:02
• I used AUC at first. But I also want to keep the original scale of probability prediction. So I'm thinking to do some post-processing based on xgboost prediction. – Salty Gold Fish Sep 7 '19 at 15:45