7
$\begingroup$

I'm using several algorithms to predict a binary target. So far I tried Gradient Boosting, Random Forest, Extra Random Trees and adaboost from scikit learn. All of these algorithms appear to predict probabilities ranging from close to zero to close to 1 with a very similar standard deviation. adaboost is the only one whose predictions are mostly compressed in the 0.4 to 0.6 range with only a minority falling outside of that range. This is not the first time I notice this behavior from this algorithm. Why is that the case? Secondly, if I wanted to blend these models (i.e average the probabilities), how would I account for the fact that adaboost probabilities's standard deviation is so different from any other algorithm? Should I rescale all predicted values from each algorithm to have similar mean and variation?

$\endgroup$

1 Answer 1

7
$\begingroup$

When using AdaBoost (and most other machine learning algorithms, such as Support Vector Machines), it is important to calibrate prediction scores. One popular method is Isotonic Regression, which I recommend for most machine learning tasks. If you pass the prediction scores from your AdaBoost model through an Isotonic Regression you'll find that it provides calibrated probabilities that range from near zero to near one. In fact, you should do this with all of the models you have mentioned before combining them in an ensemble model.

Sci-kit learn provides an Isotonic Regression function, as well as a new CalibratedClassifierCV function which will allow you to calibrate your prediction scores using cross-validation rather than holding out a separate calibration set from your training sample.

To learn more, check out these papers;

http://www.cs.cornell.edu/~caruana/niculescu.scldbst.crc.rev4.pdf

http://ijcai.org/papers13/Papers/IJCAI13-286.pdf

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.