On the "strength" of weak learners

Question

I have several closely-related questions regarding weak learners in ensemble learning (e.g. boosting).

1. This may sound dumb, but what are the benefits of using weak as opposed to strong learners? (e.g. why not boost with "strong" learning methods?)
2. Is there some sort of "optimal" strength for the weak learners (e.g. while keeping all the other ensemble parameters fixed)? Is there a "sweet spot" when it comes to their strength?
3. How can we measure the strength of a weak learner with respect to that of the resulting ensemble method. How do we quantitatively measure the marginal benefits of using an ensemble?
4. How do we compare several weak learning algorithms to decide which one to use for a given ensemble method?
5. If a given ensemble method helps weak classifiers more than strong ones, how do we tell a given classifier is already "too strong" to yield any significant gains when boosting with it?

Answer 1

This may be more in bagging spirit, but nevertheless:

• If you really have a strong learner, there is no need to improve it by any ensemble stuff.
• I would say... irrelevant. In blending and bagging trivially, in boosting making a too strong classifier may lead to some breaches in convergence (i.e. a lucky prediction may make the next iteration to predict pure noise and thus decrease performance), but this is usually repaired in proceeding iterations.

• Again, this is not the real problem. The very core of those methods is to

  1. force the partial classifiers to look deeper in the problem.
  2. join their predictions to attenuate the noise and amplify the signal.

  1) needs some attention in boosting (i.e. good boosting scheme, well behaving partial learner -- but this is mostly to be judged by experiments on the whole boost), 2) in bagging and blending (mostly how to ensure lack of correlation between learners and do not overnoise the ensemble). As long as this is OK, the accuracy of partial classifier is a third order problem.

Answer 2

First, the notions of "weak" and "strong" are only weakly defined. From my point of view they must be defined relative to the optimal Bayes classifier, which is the target of any training algorithm. With this in mind, my reply to three of the points are as follows.

1. Computational as I see it. Most weak learners I know of are computationally fast (and otherwise not worth consideration). A major point in ensemble learning is precisely that we can combine simple and fast, but not so good, learners and improve on the error rate. If we use stronger (and computationally more demanding) learners the room for improvements become smaller yet the computational cost becomes larger, which makes the use of ensemble methods less interesting. Moreover, a single strong learner may be easier to interpret. However, what is weak and what is strong depends on the problem and the optimal Bayes rate that we attempt to achieve. Hence, if a learner that is often considered strong still leaves room for improvements when boosting it and boosting is computationally feasible, then do boost ...
2. This will depend on the criteria you use to measure "optimal". In terms of error rate I would say no (I welcome any corrections if others have a different experience). In terms of speed, maybe, but I would imagine that this is highly problem dependent. I don't know any literature addressing this, sorry.
3. ?
4. Cross validation, cross validation, cross validation. Like any other comparison of methods for training with the goal of making predictions we need unbiased estimates of the generalization error for the comparison, which can be achieved by setting aside a test data set or approximating this by cross validation.