Gradient boosted decision trees can be used to solve multi-class classification problems. Friedman (2001) fit $K$ trees on each iteration—one for each class. Multiple GBM implementations also follow this practice (e.g. scikit-learn and H2O).

Instead of fitting $K$ trees on each iteration, it also seems possible to fit a single tree with $K$ outputs (it worked when I tried it on a simple, synthetic dataset). The difference with this approach is that all outputs are based on a single partition of input space. One might hope that fitting fewer trees would be computationally cheaper, but perhaps there are tradeoffs (e.g. maybe deeper trees or more iterations would be required to achieve equivalent results?)

Are there reasons to prefer $K$ single-output trees over a single tree with $K$ outputs? Do any gradient boosting implementations use the latter strategy?


Friedman (2001). Greedy function approximation: A gradient boosting machine.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.