When creating a new, $j$th learner using AdaBoost, the model for defining the weight of an example is: $$w_{j}^i = e^{-y_ih_{j-1}^i}$$ These weights are created in order that the new learner will focus on previously misclassified examples.

How does multiplying the actual and expected outputs substitute for a loss function? Would something like $e^{\epsilon_i}$($\epsilon_i = y_i - h^i_{j}$) be more appropriate?

Actually, it is natural to consider the product between predicted and expected label. It comes from the way that AdaBoost was designed.

Indeed, AdaBoost is an algorithm that aims at minimizing the exponential loss, which is defined as : $$\mathcal{l}_{exp}(h)=\mathbb{E}[\exp(-Yh(X))]$$ The question you might ask now is why it is defined this way. It comes from the convexification of the misclassification loss. Indeed, misclassification loss is classically defined as follows (in the case of binary classification with labels +1 and -1): $$\mathcal{l}(h)= \mathbb{P}(h(X) \neq Y) = \mathbb{P}(Yh(X) < 0) = \mathbb{E}[\mathbb{1}_{h(X)Y < 0}]$$ This definition is inconvenient to optimize, and we prefer considering convexified versions of the loss, defined as follows:

$$\mathcal{l}_{\phi}(h) = \mathbb{E}[\phi(Yh(X))]$$

with $\phi$ non-increasing, convex, $\phi(0) = 1$. This way we have: $$\mathcal{l}(h) \leq\mathcal{l}_{\phi}(h)$$ and it can be shown that optimizing both versions leads to the same classifier.

AdaBoost has been designed to minimize the convexified loss with $\phi(x) = \exp(-x)$.

For more details, you can refer to the first sections of the chapter Boosting and Additive Trees from the book The Elements of Statistical Learning. It is very well explained there.

• Thank you. What about if my output is continous? Commented Apr 23, 2017 at 14:09