Many people think the regression tree is only an algorithm and it doesn't make sense approach confidence interval to it so I'd like to know if there's anyone figured out how to do it.

A regression tree can be write as follows (Friedman (2000)):

$f(x)=\sum_{j=1}^{K}b_j I(x \in R_j)$

Where $R_j$ is subset of predictors train set belongs to node $j$ that is $x \in R_j$ if $x$ respect all rules that generated $j$-th node, $b_j$ is the mean of response to $x \in R_j$, $I(x \in R_j)=1$ if $x \in R_j$ and $I(x \in R_j)=0$ unlike.

A var of regression tree can be get as follows:

$V(f(x))=\sum_{j=1}^{K}V(b_j)I(x \in R_j)$ since $y_i$ is iid and $I(x \in R_j)^2=I(x \in R_j)$

$V(f(x))=\sum_{j=1}^{K}V(\frac{\sum_{i=1}^{|R_j|}y_i}{|R_j|})I(x \in R_j) $ where $|R_j|$ is $R_j's$ size.

$V(f(x))=\sum_{j=1}^{K}\frac{\sum_{i=1}^{|R_j|}V(y_i)}{|R_j|^2}I(x \in R_j)$

$V(f(x))=\sum_{j=1}^{K} \frac{\sum_{i=1}^{|R_j|}\sigma^2}{|R_j|^2}I(x \in R_j)$

$V(f(x))=\sum_{j=1}^{K}\frac{\sigma^2}{|R_j|}I(x \in R_j)$

If $y_i\sim N(u_y,\sigma^2)$ then $f(x) \sim N(\sum_{j=1}^{K}u_yI(x \in R_j), \sum_{j=1}^{K}\frac{\sigma^2}{|R_j|}I(x \in R_j))$ so we can get a confidence interval to $f(x)$ by using normal distribuition theory.


To simplify, I believe you are just saying to take the data within a node and compute an ordinary confidence interval using that data. That might work asymptotically, under some fairly strong assumptions concerning the nature of the response function. In finite samples you will have the additional problem of selection bias.

  • $\begingroup$ $|R_j|$ must be greater than 30 to asymptotically approach works $\endgroup$ – Davi Américo Jun 16 at 20:54
  • $\begingroup$ There is nothing special about the number 30. Especially given the selection bias problem, the sometimes given (and usually silly) recommendation of 30 is bound to be way too low. Again, simulation offers the best answer. $\endgroup$ – BigBendRegion Jun 17 at 0:40
  • $\begingroup$ simulation by bootstrap you mean? if you do, bootstrap only works asymptotically due to its cumulative density converges to real cumulative density. $\endgroup$ – Davi Américo Jun 17 at 1:16
  • $\begingroup$ I mean simulate to see how far off your suggested method is. You can check (i) bias, and (ii) coverage rate of the confidence intervals using simulation. That will give you an idea of how well you method works. $\endgroup$ – BigBendRegion Jun 17 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.