I am learning GBM with a focus on the interactions side of things I am aware of the H statistic which ranges from 0-1 where large values indicate strong effects. I created a dummy experiment below using R. I predict the species type from the attributes in the Iris dataset.


gbmGrid <-  expand.grid(interaction.depth = 3, 
                        n.trees = 100, 
                        shrinkage = 0.1,
                        n.minobsinnode = (3:5))

trctrl <- trainControl(method = "repeatedcv", 
                       number = 10, repeats = 3)

# 3 rows to tune

gbmModel <- train(Species~.,  data = iris,  method = "gbm",
                  trControl=trctrl, tuneGrid = gbmGrid, verbose = FALSE)  

# Find the best iteration
best.iter <- gbm.perf(gbmModel$finalModel, method="OOB")

interact.gbm(gbmModel$finalModel, data = iris, n.trees = best.iter,
             i.var = c('Petal.Length','Petal.Width','Sepal.Length'))
#      setosa   versicolor    virginica 
#0.0002366084 0.0413793842 0.0050366585 

I have two questions

  • I would just like to confirm my understanding for the final output. Is this saying that for predicting the versicolor species there is a relatively strong interaction between the Petal Length, Petal Width and Sepal.Length attributes. Comparatively this is not such an important interaction when predicting the virginica and setosa classes.
  • In my code above i have specified two items i believe there is an interaction term. Is there a way to systematically test for many interactions, say for example as an exploratory exercise (then using a test set to confirm any interactions found). I have seen the package dismo which has the function gbm.interactions but this seems to be specific to regression trees.

migrated from stackoverflow.com May 6 '18 at 17:42

This question came from our site for professional and enthusiast programmers.

  • 1
    $\begingroup$ Can you please check that you get the same results as the one I added when I ran your code? If not can you please add the H-statistics you observe? $\endgroup$ – usεr11852 May 7 '18 at 11:13
  • $\begingroup$ You should be aware that the gbm package has been orphaned. $\endgroup$ – aginensky May 7 '18 at 11:19
  • $\begingroup$ Hi @usεr11852, I can confirm your results. Thank you very much for your quick response. The interactions are small but i think for illustration they are OK. $\endgroup$ – John Smith May 7 '18 at 12:13
  • 1
    $\begingroup$ The standard replacement is xgboost, which is also supported by caret. There are two other newer packages by yandex and microsoft - if I memory serves me. They really are improvements in that they are significantly faster than gbm. $\endgroup$ – aginensky May 7 '18 at 13:28
  • 1
    $\begingroup$ Hi @aginensky, i had a look but i don't seem to be able to find anything specific to interactions. Could you point me in the right direction for xgboost; be that blogs or research papers that explore feature interactions? $\endgroup$ – John Smith May 7 '18 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.