are the marginals of boosted trees consistent if we can assume unconfoundedness?

Question

given outcome $y$ and data $X$ with data generating process $y = f(X)+\epsilon$ where $\epsilon$ independent of $X$ and gradient boosted trees as the algorithm approximating $f$, does $\partial \hat{y}/\partial{x}^* \rightarrow \partial y/\partial{x}$ as the sample size grows very large?

$^*$ when I talk about a derivative, I'm implicitly thinking of it in a "smoothed" sense. The derivatives of a tree model are always zero or undefined at the split point. I'm thinking of a grid of finite differences across a function, but I'm not quite sure how to formalize this.

Assume that turning parameters are selected by $k$-fold cross-validation (minimizing MSE in a held-out sample), including:

• number of trees
• depth of trees
• learning rate

My intuition is that it would be consistent, with the rate of convergence depending on $k$. For very large $N$ and $k$, the tuning parameters chosen cease constraining the solution. Furthermore, unless $X$ is uniformly distributed, convergence will be faster in regions where the density of the data is higher. This is because the function will not be approximated in sparse regions if the number of splits (across all trees) is finite.

But I'm just guessing! Has anyone formally proved anything?

Answer 1

A boosted tree prediction is a linear combination of regression tree predictions, which are piecewise constant functions functions of the predictors. So $\partial \hat y/\partial x$ is zero except at the steps, where it isn't defined.

Update: Results I've seen on consistency of boosted trees have been about proving the risk of the classifier converging to the best possible value (eg for convex losses). I don't know of any results on pointwise convergence.

If $\epsilon$ is independent of $x$, the best possible risk is the risk of the true value $f(X)$. If the risk of $\hat y_n$ converges to the risk of $f(X)$ under squared error loss then $\hat y$ will have to converge to $f(X)$ in mean square.

Showing that the convergence holds pointwise or, worse, uniformly on a neighbourhood of $x$, would be tricky. It's not just low-density areas that are a problem. Points close to large steps selected at early rounds might well have large error, though you could maybe fix that by sending the learning rate to zero.

So, I think it's quite likely true but hard to prove.