What’s the purpose of the 0.5 in the alpha term in Adaboost?

Question

In the AdaBoost algorithm, the contribution of each weak learner is multiplied by a weight $\alpha$ given by $$ \alpha_t = \frac{1}{2}ln\Big(\frac{1-\epsilon_t}{\epsilon_t}\Big) $$

In the Elements of Statistical Learning book, they use the following expression:

$$ \alpha_t = log\Big(\frac{1-\epsilon_t}{\epsilon_t}\Big) $$

I was wondering what is the significance of the $\frac{1}{2}$ term and would there be a difference between using log and natural log here (I assume the ESL refers to natural log when writing log)?

Answer 1

I "hate" this aspect of Adaboost because it was never really firmly formulated and different versions are presented as "AdaBoost" without much context; off the top of my head we have "AdaBoost.M1" (which sometimes is called "Discrete AdaBoost"), "AdaBoost.M2", "Real AdaBoost" and the generic "AdaBoost". Notation between ESL and Shapiro's AdaBoost paper is also inconsistent (remember $\alpha_t = 2 \beta_t$ - this will make sense later...).

What is shown in ESL is the weight of the hypothesis/classifier being computed as $\alpha_t=\text{log}(\frac{1-\epsilon_t}{\epsilon_t})$; and credit to ESL that is correct because they try to follow the AdaBoost.M1 algorithm that itself does not use the $\frac{1}{2}$ scaling; AdaBoost.M1 is presented in Freund & Schapire (1996) Experiments with a New Boosting Algorithm without a scaler of $\frac{1}{2}$ and without any reference to the exponential loss. Nevertheless, because we derive AdaBoost.M1 on the ESL using the the exponential loss directly (see Eq. 10.12 in ESL) we use "$\alpha_m = 2 \beta_m$" in the schematic of AdaBoost.M1 presented as by definition $\beta_m = \frac{1}{2} \text{log}(\frac{1-\epsilon_m}{\epsilon_m})$.

That being said, in the "original" F&S (1996) publication, we do not even have the $\log$ in the $\beta_t$ step. Taking the log happens in the final step to output the final hypothesis. And to muddle the waters a bit more, in that publication, the fraction is inverted (i.e. $\beta_t = \frac{\epsilon_t}{1-\epsilon_t}$) when first calculated and then inverted again when used in the final step to output the final hypothesis as $\text{log}(\frac{1}{\beta_t})$, i.e. in ESL the presentation of AdaBoost.M1 is slightly beautified. Is this mildly inconsistent? Yes. Is it peadagogical? Yes (I think).

Now, in regards to the $\alpha_t=\frac{1}{2}\text{log}(\frac{1-\epsilon_t}{\epsilon_t})$ in "AdaBoost". I suspect that this is from the chapter Explaining AdaBoost in the 2013 Festschrift in Honor of Vladimir N. Vapnik. That chapter's notation is misleading. That chapter itself cites the AdaBoost algorithm of Freund and Schapire (1997) A decision-theoretic generalization of on-line learning and an application to boosting which does not use the $\frac{1}{2}$ scaler for the $\beta_t$ calcuation; maybe if the text commented on the relation of $\alpha_m$ and $\beta_m$ we would know more... I hypothesise that Schapire aimed to put the exponential loss directly in $\alpha_t$ as this publication references the exponential loss' and logistic regression's connection to AdaBoost (neither of the two mid-90's publication explictily do). In a way, "this" 2013 AdaBoost is much closer to the Real AdaBoost as presented in Additive logistic regression: a statistical view of boosting Friedman et al. (2000).

To conclude: While unsatisfactory, I would suggest you focus on a "self-contained" exposition of AdaBoost as this is presented in AdaBoost.M1 within the ESL book. All other versions/papers play around with exponentiations, logs and inversions that for the most part do little to help understand the adaptive nature of boosting. The significance of the scaler $\frac{1}{2}$ is primarily theoretical showing how "faithfully" we follow the logistic loss exposition of AdaBoost as an additive model, in practice the scaler makes the "learning rate" smaller so we use smaller steps and are less prone to overfit.