I have been reading up on gradient boosting machines, and in particular GBRT's. I've come across numerous mentions (and finally tracked down some papers) on applying these models to ranking problems - particularly in web search.

Is there any particular motivation for using GBRT's for ranking problems? I assume any classifier that outputs a probability could be used for ranking problems as well. Does the flexibility in choice of loss function motivate this choice? Or is it mainly that empirical results with GBRT's greatly outperform other methods? Any insight would be appreciated.

  • $\begingroup$ While I'm not familiar with using GBRTs in ranking problems, this scikit-learn example discusses the "probability calibration" problem, and shows that Logistic Regression perform better than SVMs and Random Forests, for class probability estimates. Perhaps GBRTs should be added to that comparison as well. $\endgroup$ – shark8me Apr 28 '16 at 13:51

The Scikit learn documentation has an example of the "probability calibration" problem, which compares Logistic Regression's probability estimates against those from LinearSVC and NaiveBayes.

I added GBRT classifier to the matrix as well, and this is the corresponding graph, which shows that while the un-calibrated GBRT performs slighly poorer than Logistic Regression, the calibrated versions (using 2 different calibration methods) perform better.

Just from this experiment alone, it would be hard to make a case for GBRT over LR, however.

LR vs GBRT raw and calibrated probability

The source for my Gist which adds GBRT to the scikit-learn's original example.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.