2
$\begingroup$

I'm having trouble reproducing the results of R's gbm() function (from the gbm package) on a minimal example. What am I doing wrong?

library(gbm)  # http://www.saedsayad.com/docs/gbm2.pdf

train <- data.frame(
  X1 = factor(c("A", "A", "A", "B", "B")),
  X2 = factor(c("A", "A", "B", "B", "B")),
  Y = c(0, 1, 3, 4, 7)
)
train
  X1 X2 Y
1  A  A 0
2  A  A 1
3  A  B 3
4  B  B 4
5  B  B 7

I want to minimize absolute deviation using trees of depth 1. The initialization step should predict the median of $Y$ for all samples which is 3.

# (0 rounds, mae)
model.gbm <- gbm(
  Y ~ X1 + X2, data = train, distribution="laplace", n.tree = 0, shrinkage = 1, n.minobsinnode=1, bag.fraction=1, 
  interaction.depth = 1, verbose=TRUE
)
train$Pred.mae.gbm0 <- predict(model.gbm, newdata=train, n.trees=model.gbm$n.trees)
train
  X1 X2 Y Pred.mae.gbm0
1  A  A 0             3
2  A  A 1             3
3  A  B 3             3
4  B  B 4             3
5  B  B 7             3

Check.

According to Wikipedia I need to calculate the pseudo-residuals

$ r_{im} = -\left[\frac{\partial L(y_i, F(x_i))}{\partial F(x_i)}\right]_{F(x)=F_{m-1}(x)} \quad \mbox{for } i=1,\ldots,n. $

For absolute deviation, the negative gradient of $L$ w.r.t. $F$ is simply $sign(y_i - \hat{y}_i)$. So we need to fit a regression tree to

  X1 X2 NegGrad
1  A  A      -1
2  A  A      -1
3  A  B       0
4  B  B       1
5  B  B       1

Splitting on X1 or X2 should give equivalent results. Let's assume we split on X1 such that the left node contains {-1, -1, 0} and the right node contains {1, 1}. (So, the predictions are -1 and 1 respectively, since MAE uses median to make terminal node predictions?)

Now we update the model using $F_{m}(x)=F_{{m-1}}(x)+\gamma _{m}h_{m}(x)$ where $\gamma _{m}$ is the step size we choose (in this case, 1). Let's look at what gbm() produces here

# (1 round, mae)
model.gbm <- gbm(
  Y ~ X1 + X2, data = train, distribution="laplace", n.tree = 1, shrinkage = 1, n.minobsinnode=1, bag.fraction=1, 
  interaction.depth = 1, verbose=TRUE
)
train$Pred.mae.gbm1 <- predict(model.gbm, newdata=train, n.trees=model.gbm$n.trees)
  X1 X2 Y Pred.mae.gbm0 Pred.mae.gbm1
1  A  A 0             3           1.0
2  A  A 1             3           1.0
3  A  B 3             3           1.0
4  B  B 4             3           5.5
5  B  B 7             3           5.5

How does the prediction for rows 1 go from 3 in round0 to 1 in round1 if the step size is only 1 and the negative gradient is just -1 for the first base learner?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.