# Why is boosting effective?

Boosting takes a bunch of weak learners and creates a strong learner. But why is it so difficult to create a strong learner right from the beginning without using boosting techniques? And therefore remove the need for boosting.

• I'm not sure this question makes sense... For instance, I could take a bunch of weak learners, use a boosting algorithm to weight them, then "rip out" the boosting code and make one mega-classifier that performs identically to the ensemble. So... what are you asking exactly? – Stumpy Joe Pete Jul 3 '13 at 18:55

In most practical cases, one can do as you say. This is because most predictive models (such as the Gail model for breast cancer risk) only use a handful of extremely prognostic markers. That is to say we obtain predictors with very high recall having target parameter space of dimension $q \ll p$ (we do well with only a few predictors, and throw the rest away).
1. $p \gg n$: right off the bat, this excludes many traditional methods. With the parametric modeling approach, forward stepwise model selection, and even LASSO that means you can have at most $n$ predictors in the model and they will be biased.
2. $|w_j| < \epsilon$ , $j = \{1, 2, \ldots p\}$ for $\epsilon$ small. We might even state it in such a convergence criterion as $n ||w||_2 \rightarrow d$, $d$ finite. i.e. we have really weak effects. Methods which are equivalent to tests of hypothesis in which $\mathcal{H}_0: w_j = 0$ will in general have too large a type II error rate for our purposes. This includes LASSO as I said.