7
$\begingroup$

I can't figure out what does learning_rate stand for in sklearn implementation of Adaboost. When i see the original algorithm i don't see any "learning_rate"...

Meanwhile i can see from https://fr.wikipedia.org/wiki/AdaBoost that the training errors are weighted thanks to $D_t(i)$ (where $i$ is attached to the $i$th training instance in the training matrix $X$). Is there any relation between the sklearn "learning_rate" and this $D_t$ ?

$\endgroup$
4

2 Answers 2

5
$\begingroup$

The official documentation states that "The learning rate shrinks the contribution of each regressor by learning_rate.". Thus, basically we need to understand three concepts:

1. Weak Classifier

A model whose error rate is only slightly better than random guessing, that is to say, 50% accuracy.

2. Boosting

This technique has the objective to apply $K$ different times (sequentially) a model to modified versions of the data. So, suppose at each iteration $i \in \{1,2,..., K\}$ you build a new tree model $T_{i}$

\begin{align} T_{i+1}(x) = T_{i}(x) + \alpha M(x), \end{align}

where $$M(x) = \sum_{j=1}^{J} t(x, \theta_{j})$$ is the sum of trees with different paramaters $\theta_{j}$ and $\alpha$ is the learning rate between 0 and 1.

3. Learning Rate

This parameter controls how much I'm going to contribute with the new model to the existing one. Normally there is trade off between the number of iterations $K$ and the value of $\alpha$. In other words, when taking smaller values of alpha ($\alpha \approx 0$) you should consider more $K$ iterations, so that your base model (the weak classifier) continues to improve. According to Jerome Friedman, it is suggested to set $\alpha$ to smaller values ($\alpha < .1$).

$\endgroup$
2
  • 1
    $\begingroup$ you saved my search. Thank you. $\endgroup$
    – arilwan
    Jun 15, 2021 at 8:55
  • $\begingroup$ Do you happen to have a source for this? Your explanation makes sense and is very clear, but I'm concerned about the discrepancy with sklearn docs that say LR is between 0 and inf, not 0 and 1, and that higher LR increases contribution of each classifier - opposite of what you wrote. $\endgroup$
    – Tim
    Sep 24 at 2:05
0
$\begingroup$

It looks to me that this is mainly the problem with sklearn docs. Observe in a 2021 GitHub issue about this exact question, not one person has referenced a paper that is actually implemented, and entire discussion is purely about style.

I think @XavierBourretSicotte's answer above should be the accepted one, because it is consistent with sklearn's implementation. I'll also reopen that issue and suggest a link to the paper and/or precise definition of the LR.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.