Boosting takes a bunch of weak learners and creates a strong learner. But why is it so difficult to create a strong learner right from the beginning without using boosting techniques? And therefore remove the need for boosting.

  • $\begingroup$ I'm not sure this question makes sense... For instance, I could take a bunch of weak learners, use a boosting algorithm to weight them, then "rip out" the boosting code and make one mega-classifier that performs identically to the ensemble. So... what are you asking exactly? $\endgroup$ – Stumpy Joe Pete Jul 3 '13 at 18:55

In most practical cases, one can do as you say. This is because most predictive models (such as the Gail model for breast cancer risk) only use a handful of extremely prognostic markers. That is to say we obtain predictors with very high recall having target parameter space of dimension $q \ll p$ (we do well with only a few predictors, and throw the rest away).

But in absence of a "few good markers" and with an abundance of weak ones, no specialized technique exists outside of boosting which can give you low bias and good prediction. Why is that unique?

  1. $p \gg n$: right off the bat, this excludes many traditional methods. With the parametric modeling approach, forward stepwise model selection, and even LASSO that means you can have at most $n$ predictors in the model and they will be biased.

  2. $|w_j| < \epsilon$ , $j = \{1, 2, \ldots p\}$ for $\epsilon$ small. We might even state it in such a convergence criterion as $n ||w||_2 \rightarrow d$, $d$ finite. i.e. we have really weak effects. Methods which are equivalent to tests of hypothesis in which $\mathcal{H}_0: w_j = 0$ will in general have too large a type II error rate for our purposes. This includes LASSO as I said.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.