Which optimization technique does XGBoost use? I have trained my XGBoost classifier using the 'binary:logistic' objective. Does it use SGD / Adam?
XGBoost uses a tree based boosting algorithm.
I'll highlight a few key aspects of this package based on the author's paper "XGBoost: A Scalable Tree Boosting System" as follows:
- Its learning objective function is a set of additive regression trees (refer to section 2.1 of the paper).
- Each additive tree contains leaf weights with continuous scores. These are used to classify an example by summing up the scores in the corresponding leaves.
- The objective function to be minimized is a regularised differentiable convex loss function that measures the difference between the prediction and the target (as given in equation 2).
- As described in this paper (section 2.2), the tree ensemble model in equation 2 includes functions as parameters and cannot be optimized using traditional optimization methods in Euclidean space. Instead, the model is trained in an additive manner.
You should refer to the paper for a more complete description of their method, the greedy search algorithm they've implemented and the improvements they've made which makes this implementation work so well.