I'm looking for work done on showing the consistency of gradient boosting methods such as gradient boosted decision trees. Friedman's original work only introduces the algorithm, but does not provide any analysis of the consistency properties.

By "consistency" I mean in the sense that following the gradient boosting algorithm minimizes the expected loss as the number of samples goes to infinity (https://en.wikipedia.org/wiki/Consistency_(statistics)).

I'm also interested in convergence rate bounds under different assumptions on the distribution.

My expectation is that the results from empirical risk minimization and analyses of gradient descent apply directly to gradient boosting, but I'm wondering if it's more subtle than that. Are there sources discussing this in more detail?


1 Answer 1


[1] Shows consistency of the gradient boosting algorithm and gives some background on related work.

[2] Extends gradient boosting to the case of nesterov momentum and gives bounds on the optimization rate (how fast does the excess empirical risk go to zero).


[1] Biau, G., Cadre, B. (2017). Optimization by gradient boosting arXiv https://arxiv.org/abs/1707.05023

[2] Lu, H., Karimireddy, S., Ponomareva, N., Mirrokni, V. (2019). Accelerating Gradient Boosting Machine arXiv https://arxiv.org/abs/1903.08708


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.