How to find optimal values for the tuning parameters in boosting trees ?

Question

I realise that there are 3 tuning parameters in the boosting trees model, i.e.

1. the number of trees (number of iterations)
2. shrinkage parameter
3. number of splits (size of each constituent trees)

My question is: for each of the tuning parameters, how should I find its optimal value ? And what method ?

Note that: the shrinkage parameter and the number of trees parameter operate together, i.e. a smaller value for shrinkage parameter leads to a higher value for the number of trees. And we need to take this into account too.

I am particularly interested in the method to find the optimal value for the number of splits. Should it be based on cross-validation or domain knowledge about the model behind?

And how are these things carried out in the gbm package in R ?

Answer 1

The caret package in R is tailor made for this.

Its train function takes a grid of parameter values and evaluates the performance using various flavors of cross-validation or the bootstrap. The package author has written a book, Applied predictive modeling, which is highly recommended. 5 repeats of 10-fold cross-validation is used throughout the book.

For choosing the tree depth, I would first go for subject matter knowledge about the problem, i.e. if you do not expect any interactions - restrict the depth to 1 or go for a flexible parametric model (which is much easier to understand and interpret). That being said, I often find myself tuning the tree depth as subject matter knowledge is often very limited.

I think the gbm package tunes the number of trees for fixed values of the tree depth and shrinkage.

Answer 2

There are two good sources for the boosted regression trees and gbm package. For the explanation of BRT and the optimization of the number of trees (nt), learning rate (lr) and tree complexity (tc) see A working guide to boosted regression trees Although it is focusing on ecology I think you wilĺ not find a better introduction to BRT.

For the implementation of BRT in the gbm package, see Boosted Regression Trees for ecological modeling

In short, a rule of thumb is to select a learning rate that allows the BRT model to fit at least 1000 trees, so propably you will need a low learning rate, maybe 0.001 to accomplish that. But it depends on the size of your data, see fig. 2 and 3 in the Working guide to BRT. I think one possible way would be the set up different models in BRT according to you data size, for example combine different lr (0.1, 0.01, 0.001), tc (1, 3, 5, 7, 9, 20) with different bag.fractions(0.5, 0.7, 0.9) and choose the best one according to lowest deviance or highest ROC score. Maybe it helped.