3
$\begingroup$

I have trouble understanding how relative influence of a variable is calculated in a boosted regression tree. I am reading from the following paper by Friedman and Meulman.

Multiple additive regression trees with application in epidemiology http://onlinelibrary.wiley.com/doi/10.1002/sim.1501/pdf

"The relative contribution of any one explanatory variable ($x_j$) is based on how often it is selected to split individual trees, weighted by the squared improvement to the model ($I_j^2$) resulting from the sum of these trees (i.e. from $m = 1$ to $M$ the total number of trees):

$$\hat I_j^2 = \frac{1}{M} \sum_{m=1}^M I_j^2(Tm)$$

where $I_j^2$ is the relative influence of input variable $j$ for individual tree $Tm$

I do not understand how the term $I_j^2$ (which is the squared improvement of the model) is calculated for each tree. Can anyone please explain me this.

Improve this question
$\endgroup$
4
  • $\begingroup$ which text are you talking about? $\endgroup$ – Haitao Du Aug 31 '16 at 14:48
  • $\begingroup$ Possible duplicate: stats.stackexchange.com/questions/162162/… $\endgroup$ – Matthew Drury Aug 31 '16 at 15:02
  • $\begingroup$ Thanks. I was looking for an explanation which is opposite to what is given in the link you provided. I am looking for a simpler explanation (something like back of the envelope calculation) whereas the link gave a bit code-like explanation which goes above head. Sorry :( $\endgroup$ – user53020 Aug 31 '16 at 15:47
  • $\begingroup$ No problem, fair enough. I'll try to add an answer like that later in the day. $\endgroup$ – Matthew Drury Aug 31 '16 at 16:47
1
$\begingroup$

This is what the paper says:

"As noted above, all of the input predictor variables are seldom equally relevant for prediction. Often only a few of them have substantial influence on the response; the vast majority are irrelevant and could just as well have not been measured. It is often useful to learn the relative importance or contribution of each input variable in predicting the response. For a single tree T, Breiman et al. [1] proposed a measure of (squared) relevance of your measure for each predictor variable xj, based on the number of times that variable was selected for splitting in the tree weighted by the squared improvement to the model as a result of each of those splits. This importance measure is easily generalized to additive tree expansions (3); it is simply averaged over the trees."

So if you can get the estimate, then you average it over all trees. So how many time was it used in a tree and its influence on improvement, depending on how that is being measured (e.g., accuracy). Have you had a look at the Breiman paper?

Breiman L, Friedman JH, Olshen R, Stone C. Classication and Regression Trees . Wadsworth: Pacic Grove, 1984

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks. i read this paper. This is exactly where I get confused. How is "(squared) relevance" in line 5 calculated or what does it mean? $\endgroup$ – user53020 Sep 2 '16 at 21:47
  • $\begingroup$ Whatever you do, be sure to get an uncertainty interval for the variable importance measures. You may be surprised at how unstable these measures are. See onlinelibrary.wiley.com/doi/abs/10.1002/sim.7803 for example. $\endgroup$ – Frank Harrell Jun 10 '18 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.