given outcome $y$ and data $X$ with data generating process $y = f(X)+\epsilon$ where $\epsilon$ independent of $X$ and gradient boosted trees as the algorithm approximating $f$, does $\partial \hat{y}/\partial{x}^* \rightarrow \partial y/\partial{x}$ as the sample size grows very large?

$^*$ when I talk about a derivative, I'm implicitly thinking of it in a "smoothed" sense. The derivatives of a tree model are always zero or undefined at the split point. I'm thinking of a grid of finite differences across a function, but I'm not quite sure how to formalize this.

Assume that turning parameters are selected by $k$-fold cross-validation (minimizing MSE in a held-out sample), including:

  • number of trees
  • depth of trees
  • learning rate

My intuition is that it would be consistent, with the rate of convergence depending on $k$. For very large $N$ and $k$, the tuning parameters chosen cease constraining the solution. Furthermore, unless $X$ is uniformly distributed, convergence will be faster in regions where the density of the data is higher. This is because the function will not be approximated in sparse regions if the number of splits (across all trees) is finite.

But I'm just guessing! Has anyone formally proved anything?


1 Answer 1


A boosted tree prediction is a linear combination of regression tree predictions, which are piecewise constant functions functions of the predictors. So $\partial \hat y/\partial x$ is zero except at the steps, where it isn't defined.

Update: Results I've seen on consistency of boosted trees have been about proving the risk of the classifier converging to the best possible value (eg for convex losses). I don't know of any results on pointwise convergence.

If $\epsilon$ is independent of $x$, the best possible risk is the risk of the true value $f(X)$. If the risk of $\hat y_n$ converges to the risk of $f(X)$ under squared error loss then $\hat y$ will have to converge to $f(X)$ in mean square.

Showing that the convergence holds pointwise or, worse, uniformly on a neighbourhood of $x$, would be tricky. It's not just low-density areas that are a problem. Points close to large steps selected at early rounds might well have large error, though you could maybe fix that by sending the learning rate to zero.

So, I think it's quite likely true but hard to prove.

  • $\begingroup$ You're obviously correct. Implicitly I'm talking about some kind of smoother. How do I make this explicit..? $\endgroup$ Jan 26 at 21:13
  • $\begingroup$ THough I suppose that as the number of trees goes to infinity and if the data is from a purely-continuous distribution, you'd asymptote very slowly to a split point everywhere. But this is a tangent, and I think that you get the sense of my question. Are booster marginals biased in large samples near the center of the support of the data? $\endgroup$ Jan 26 at 21:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.