11
$\begingroup$

Is there a strategy for choosing the number of trees in a GBM? Specifically, the ntrees argument in R's gbm function.

I don't see why you shouldn't set ntrees to the highest reasonable value. I've noticed that a larger number of trees clearly reduces the variability of results from multiple GBMs. I don't think that a high number of trees would lead to overfitting.

Any thoughts?

$\endgroup$
3
$\begingroup$

This is GBM:

"I don't think that ... " has been the dangerous first part of many sentences.

Good enough is meaningless without a measure of goodness, a rubric.

What are the measures of goodness for any other method?

  • Difference between model and data (sse, ...)
  • Divergence of Error in a holdout set (training error vs. test error)
  • Parameter count to sample count ratio (most folks like 5 samples per parameter or 30 samples per parameter)
  • Cross validation (ensemble methods on divergence of error tests)

Like a neural network, or spline, you can perform piecewise linear interpolation on the data and get a model that cannot generalize. You need to give up some of the "low error" in exchange for general applicability - generalization.

More links:

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

I did find some insight into the problem: http://cran.r-project.org/web/packages/dismo/vignettes/brt.pdf

The gbm.step function can be used to determine the optimal number of trees. I'm still not sure what causes model deviance to increase after a certain number of trees, so I'm still willing to accept a response that answers this part of the question!

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Overfitting causes the increase. Most good methods make a holdout set, and use it to test the model, but not to update the model. This allows detection of onset of overfit. $\endgroup$ – EngrStudent Jan 26 '14 at 20:59
0
$\begingroup$

This is the working guid to boosted regression trees from Elith et al.: http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2656.2008.01390.x/full Very helpful!

You should at least use 1000 trees. As far as I understood, you should use the combination of learning rate, tree complexity and number of trees that achieves the minumum predictive error. Smaller values of the learning rate lead to larger training risk for the same number of iterations, while each iteration reduces the training risk. If the number of trees is large enough, the risk can be made arbitrarily small (see: Hastie et al., 2001, "The Elements of Statistical Learning, Data Mining, Inference and Prediction").

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ It's true that that Elith et al. suggest as a rule of thumb to use 1000 trees. However, this is based on a detailed analysis of predictive stability for the specific dataset used in the paper. It seems unlikely that the same number would work for any possible dataset. Maybe you could expand your answer a bit by giving some details on the analysis they performed, particularly in Appendix S1. $\endgroup$ – DeltaIV Jan 25 '17 at 10:37
0
$\begingroup$

As common in some machine learning algorithms, Boosting is subject to Bias-variance trade-off regarding number of trees. Loosely speaking, this trade-off tells you that: (i) weak models tend to have high bias and low variance: they are too rigid to capture variability in the training dataset, so will not perform well in the test set either (high test error) (ii) very strong models tend to have low bias and high variance: they are too flexible and they overfit the training set, so in the test set (as the datapoints are different from the training set) they will also not perform well (high test error)

The concept of Boosting trees is to start with shallow trees (weak models) and keep adding more shallow trees that try to correct previous trees weakenesses. As you do this process, the test error tends to go down (because the overall model gets more flexible/powerful). However, if you add too many of those trees, you start overfitting the training data and therefore test error increases. Cross-validation helps with finding the sweet spot

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.