33
$\begingroup$

I had a question on the interaction depth parameter in gbm in R. This may be a noob question, for which I apologize, but how does the parameter, which I believe denotes the number of terminal nodes in a tree, basically indicate X-way interaction among the predictors? Just trying to understand how that works. Additionally, I get pretty different models if I have a dataset with say two different factor variables versus the same dataset except those two factor variables are combined into a single factor (e.g. X levels in factor 1, Y levels in factor 2, combined variable has X * Y factors). The latter is significantly more predictive than the former. I had thought increasing interaction depth would pick this relationship up.

Improve this question
$\endgroup$

5 Answers 5

Reset to default
25
$\begingroup$

Both of the previous answers are wrong. Package GBM uses interaction.depth parameter as a number of splits it has to perform on a tree (starting from a single node). As each split increases the total number of nodes by 3 and number of terminal nodes by 2 (node $\to$ {left node, right node, NA node}) the total number of nodes in the tree will be $3*N+1$ and the number of terminal nodes $2*N+1$. This can be verified by having a look at the output of pretty.gbm.tree function.

The behaviour is rather misleading, as the user indeed expects the depth to be the depth of the resulting tree. It is not.

Improve this answer
$\endgroup$
4
  • $\begingroup$ What is N here: Numer of nodes, interaction.depth or something else? $\endgroup$
    – Funkwecker
    Jun 26, 2017 at 6:20
  • $\begingroup$ It is a number of splits performed, starting from a single node (also, interaction depth). $\endgroup$
    – random
    Jun 26, 2017 at 20:35
  • 1
    $\begingroup$ I think each split only increases the total number of terminal nodes by 1. so suppose a tree just have only one split, then it has 2 terminal nodes, now you perform a split on one of the previous terminal nodes, and then there's 3 terminal nodes now. so the increment is just 1. Do I get this right or I mis-understood something? $\endgroup$
    – Lily Long
    Aug 15, 2017 at 18:57
  • 1
    $\begingroup$ @LilyLong It might not be immediately clear, but gbm actually splits nodes in three, the third child grouping NA values (i.e. those that cannot be directly compared to the given value). That means that each split increases number of nodes by two. The package might have evolved since I have last used it to avoid creating this third child, so please double-check this by running pretty.gbm.tree function. $\endgroup$
    – random
    Aug 16, 2017 at 19:58
2
$\begingroup$

I had a question on the interaction depth parameter in gbm in R. This may be a noob question, for which I apologize, but how does the parameter, which I believe denotes the number of terminal nodes in a tree, basically indicate X-way interaction among the predictors?

Link between interaction.depth and the number of terminal nodes

One as to see interaction.depth as the number of split nodes. An interaction.depth fixed at k will result in nodes with k+1 terminal nodes (omitting the NA nodes), so we have : $$interaction.depth=\#\{Terminal Nodes\}+1 $$

Link between interaction.depth and the interaction order

The link between interaction.depth and interaction order is more tedious.

Instead of reasoning with the interaction.depth, let's reason with the number of terminal nodes, which we will called J.

Example: Let's say you have J=4 terminal nodes (interaction.depth=3) you can either :

  1. do the first split on the root, then the second split on the left node of the root and the third split on the right node of the root. The interaction order for this tree will be 2.
  2. do the first split on the root, then the second split on the left (respectively right) node of the root, and a third split on this very left (respectively right) node. The interaction order for this tree will be 3.

So you cannot know in advance what will be the interaction order between your features in a given tree. However it is possible to upper bound this value. Let P be the interaction order of the features in a given tree. We have : $$P\leq min(J-1,n)$$ with n being the number of observations. For more details see the section 7 of the original article of Friedman.

Improve this answer
$\endgroup$
1
$\begingroup$

Previous answer is not correct.

Stumps will have an interaction.depth of 1 (and have two leaves). But interaction.depth=2 gives three leaves.

So: NumberOfLeaves = interaction.depth + 1

Improve this answer
$\endgroup$
0
$\begingroup$

Actually, the previous answers are incorrect.

Let K be the interaction.depth, then the number of nodes N and leaves L (i.e terminal nodes) are respectively given by the following: $$\begin{align*} N &= 2^{(K+1)} - 1\\ L &= 2^K \end{align*} $$ The previous 2 formulas can easily be demonstrated: a tree of depth K can be seen as having K+1 levels k ranging from 0 (root level) to K (leaf level).

Each of these levels has $2^k$ nodes. And the tree's total number of nodes is the sum of the number of nodes at each level.

In mathematical terms: $$ N = \sum_{k=0}^K 2^k) $$

which is equivalent to:

$$N = 2^{(K+1)} - 1 $$(as per the formula of the sum of the terms of a geometrical progression).

Improve this answer
$\endgroup$
0
$\begingroup$

You can try

table(predict(gbm( y ~.,data=TrainingData, distribution="gaussian", verbose =FALSE, n.trees =1 , shrinkage =0.01, bag.fraction =1 , interaction.depth = 1 ),n.trees=1))

and see that there are only 2 unique predicted values. interaction.depth = 2 will get you 3 distinct predicted values. And convince yourself.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Not clear how this answers the question. $\endgroup$
    – Michael R. Chernick
    Jun 21, 2018 at 19:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.