6
$\begingroup$

The boosting algorithm Adaboost (when using a tree) has three core parameters:

  • number of weak learners to train
  • learning rate
  • max nb of splits (depth of tree)

What are good practices, perhaps proven empirically, for finding the appropriate value for these parameters?

One option I can think of is to do a grid search via cross validation and get out the validation accuracy (1-generalization error).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

Number of weak learners

Train many, many weak learners. Then look at a test-error vs. number of estimators curve to find the optimal number.

Learning rate

Smaller is better, but you will have to fit more weak learners the smaller the learning rate. During initial modeling and EDA, set the learning rate rather large (0.01 for example). Then when fitting your final model, set it very small (0.0001 for example), fit many, many weak learners, and run the model over night.

Maximum number of splits

There is no a-priori best answer. You need to grid search this one.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks. Sounds a lot like grid search. Any literature/references you can think of that discuss this topic? $\endgroup$
    – WJA
    Sep 21, 2017 at 7:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.