4
$\begingroup$

The boosting algorithm Adaboost (when using a tree) has three core parameters:

  • number of weak learners to train
  • learning rate
  • max nb of splits (depth of tree)

What are good practices, perhaps proven empirically, for finding the appropriate value for these parameters?

One option I can think of is to do a grid search via cross validation and get out the validation accuracy (1-generalization error).

$\endgroup$
5
$\begingroup$

Number of weak learners

Train many, many weak learners. Then look at a test-error vs. number of estimators curve to find the optimal number.

Learning rate

Smaller is better, but you will have to fit more weak learners the smaller the learning rate. During initial modeling and EDA, set the learning rate rather large (0.01 for example). Then when fitting your final model, set it very small (0.0001 for example), fit many, many weak learners, and run the model over night.

Maximum number of splits

There is no a-priori best answer. You need to grid search this one.

$\endgroup$
  • $\begingroup$ Thanks. Sounds a lot like grid search. Any literature/references you can think of that discuss this topic? $\endgroup$ – JohnAndrews Sep 21 '17 at 7:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.