1
$\begingroup$

I saw the following in a Random Forrest calculation. My understanding of logarithms is not intuitive, I always have to look them up. It was asked:

Calculate this decision tree’s weight in the ensemble
the weight of this tree = learning rate * log( (1 — e) / e)

It was in middle of an Adaboost tutorial. My question is why is this equation with the log chosen? What special properties does it have that makes a good constant for the learning rate?

Basic Ensemble Learning (Random Forest, AdaBoost, Gradient Boosting)- Step by Step Explained Under the section Adaboost.

$\endgroup$
1
  • 3
    $\begingroup$ Adaboost and random forest are different algorithms, random forest doesn't have learning rate. $\endgroup$
    – Tim
    Jun 5, 2019 at 3:40

1 Answer 1

2
$\begingroup$

First things first: As Tim mentioned (+1), there is no learning rate concept within the Random Forest framework. Adaboost and Gradient boosting are primarily boosting algorithms that "learn from their mistakes", while Random Forest is primarily a bagging (bootstrap aggregation) algorithm that "learns by model averaging".

Some context for the learning rate before we move forward: Within boosting each iteration (hopefully) allows us to improve on our training loss. These improvements though are scaled by the learning rates as to make smaller steps in the function domain our model resides. In practical terms, we perform smaller updates to avoid overfitting our data.

Now regarding AdaBoost using the log ratio of the (sum of the) errors in its weight updates: The error term $e_t$ is the sum of the weight $w$ of the misclassified points. These weights $w$ themselves are scaled such that they sum up to $1$. i.e. the term $e$ will never be larger than $1$. Now, when $e_t$ is very small, i.e. the tree is a good classifier, AdaBoost wants that tree to be assigned more weight than if it were a bad classifier. For example, assuming we have a strong classifier with $e_t \leq 0.01$ the corresponding log ratio term equals $\log(99)$ (around $4.6$), while if $e_t \approx 0.5$ the corresponding ratio is approaches $\log(1)$, i.e $0$. We need to note here that practically $e_t$ is assumed to never get above $0.5$; if a base classifier had a $e_t$ above $0.5$, we would invert the signs of that base classifier and get another with a smaller $e_t$. And this brings us to why we picked the term $\log(\frac{1-e_t}{e_t})$ as our scaler for the learning rate: it allows us to adapt the learning rate per base classifier. If the base classifier tree is strong (i.e. $e_t$ is low) then a large step is taken. If the base classifier tree is bad (i.e. $e_t$ approaches $0.5$) then a much smaller step is taken. With each step we update our predictions; this means that a "large step" potentially changes our predictions for the better while a "small step" does not significantly deteriorate our predictions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.