22
$\begingroup$

I have several closely-related questions regarding weak learners in ensemble learning (e.g. boosting).

  1. This may sound dumb, but what are the benefits of using weak as opposed to strong learners? (e.g. why not boost with "strong" learning methods?)
  2. Is there some sort of "optimal" strength for the weak learners (e.g. while keeping all the other ensemble parameters fixed)? Is there a "sweet spot" when it comes to their strength?
  3. How can we measure the strength of a weak learner with respect to that of the resulting ensemble method. How do we quantitatively measure the marginal benefits of using an ensemble?
  4. How do we compare several weak learning algorithms to decide which one to use for a given ensemble method?
  5. If a given ensemble method helps weak classifiers more than strong ones, how do we tell a given classifier is already "too strong" to yield any significant gains when boosting with it?
$\endgroup$
16
$\begingroup$

This may be more in bagging spirit, but nevertheless:

  • If you really have a strong learner, there is no need to improve it by any ensemble stuff.
  • I would say... irrelevant. In blending and bagging trivially, in boosting making a too strong classifier may lead to some breaches in convergence (i.e. a lucky prediction may make the next iteration to predict pure noise and thus decrease performance), but this is usually repaired in proceeding iterations.
  • Again, this is not the real problem. The very core of those methods is to

    1. force the partial classifiers to look deeper in the problem.
    2. join their predictions to attenuate the noise and amplify the signal.

    1) needs some attention in boosting (i.e. good boosting scheme, well behaving partial learner -- but this is mostly to be judged by experiments on the whole boost), 2) in bagging and blending (mostly how to ensure lack of correlation between learners and do not overnoise the ensemble). As long as this is OK, the accuracy of partial classifier is a third order problem.

$\endgroup$
  • $\begingroup$ Thanks @mbq. Does the above mean that weak classifiers typically benefit more from ensemble methods than strong ones? (i.e. boosting helps weak classifiers more than strong ones). In this sense, how do we know a given classifier is already strong enough for a certain ensemble method? (e.g. how can you roughly tell you have a strong learner that won't benefit much from boosting?) $\endgroup$ – Amelio Vazquez-Reina Jul 29 '11 at 12:32
  • 1
    $\begingroup$ Rather only weak classifiers give a space for improvement. In general strength is an abstract quality and we cannot really measure it. The only certain test is just to make an experiment and check whether ensembing significantly increases performance. If so, classifier was weak. If no, well, we still know nothing. $\endgroup$ – user88 Jul 29 '11 at 13:41
11
$\begingroup$

First, the notions of "weak" and "strong" are only weakly defined. From my point of view they must be defined relative to the optimal Bayes classifier, which is the target of any training algorithm. With this in mind, my reply to three of the points are as follows.

  1. Computational as I see it. Most weak learners I know of are computationally fast (and otherwise not worth consideration). A major point in ensemble learning is precisely that we can combine simple and fast, but not so good, learners and improve on the error rate. If we use stronger (and computationally more demanding) learners the room for improvements become smaller yet the computational cost becomes larger, which makes the use of ensemble methods less interesting. Moreover, a single strong learner may be easier to interpret. However, what is weak and what is strong depends on the problem and the optimal Bayes rate that we attempt to achieve. Hence, if a learner that is often considered strong still leaves room for improvements when boosting it and boosting is computationally feasible, then do boost ...
  2. This will depend on the criteria you use to measure "optimal". In terms of error rate I would say no (I welcome any corrections if others have a different experience). In terms of speed, maybe, but I would imagine that this is highly problem dependent. I don't know any literature addressing this, sorry.
  3. ?
  4. Cross validation, cross validation, cross validation. Like any other comparison of methods for training with the goal of making predictions we need unbiased estimates of the generalization error for the comparison, which can be achieved by setting aside a test data set or approximating this by cross validation.
$\endgroup$
  • $\begingroup$ Thanks @NRH, that's very helpful. I have separated the third question into two separate questions, since I think they probably require different answers. $\endgroup$ – Amelio Vazquez-Reina Jul 28 '11 at 21:39
  • $\begingroup$ So is there a way to find out how close a classifier is to the optimal Bayes classifier? If it is already close enough then we can't improve it. $\endgroup$ – highBandWidth Nov 5 '15 at 23:43
  • $\begingroup$ @highBandWidth, it is not possible to know what the Bayes rate is. It is a theoretical quantity that relies on the unknown distribution. Theoretical assumptions might provide lower and upper (asymptotic) bounds, and by using cross validation or independent test data it is possible to accurately estimate upper bounds. But unless you know the distribution, it is impossible to tell if such upper bounds are tight or leave room for improvement. $\endgroup$ – NRH Nov 6 '15 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.