# What is meant by 'weak learner'?

Can anyone tell me what is meant by the phrase 'weak learner'? Is it supposed to be a weak hypothesis? I am confused about the relationship between a weak learner and a weak classifier. Are both the same or is there some difference?

In the adaboost algorithm, T=10. What is meant by that? Why do we select T=10?

• Welcome to the site, @vrushali. I edited this to make the English smoother; please make sure it still says what you mean. Also, I am not sure if the second set of questions (about adaboost) is the same as the first set of questions; it may make more sense to separate them into different threads. – gung Jan 13 '14 at 4:34

A 'weak' learner (classifer, predictor, etc) is just one which performs relatively poorly--its accuracy is above chance, but just barely. There is often, but not always, the added implication that it is computationally simple. Weak learner also suggests that many instances of the algorithm are being pooled (via boosting, bagging, etc) together into to create a "strong" ensemble classifier.

It's mentioned in the original AdaBoost paper by Freund & Schapire:

Perhaps the most surprising of these applications is the derivation of a new application for "boosting", i.e., converting a "weak" PAC learning algorithm that performs just slightly better than random guessing into one with arbitrarily high accuracy. --(Freund & Schapire, 1995)

but I think the phrase is actually older than that--I've seen people cite a term paper(?!) by Michael Kearns from the 1980s.

The classic example of a Weak Learner is a Decision Stump, a one-level decision tree (1R or OneR is another commonly-used weak learner; it's fairly similar). It would be somewhat strange to call a SVM a 'weak learner', even in situations where it performs poorly, but it would be perfectly reasonable to call a single decision stump a weak learner even when it performs surprisingly well by itself.

Adaboost is an iterative algorithm and $T$ typically denotes the number of iterations or "rounds". The algorithm starts by training/testing a weak learner on the data, weighting each example equally. The examples which are misclassified get their weights increased for the next round(s), while those that are correctly classified get their weights decreased.

I'm not sure there's anything magical about $T=10$. In the 1995 paper, $T$ is given as a free parameter (i.e., you set it yourself).

• As far as I know a DecisionStump is different from 1Rule. A Decision Stump is always a binary 1-level tree (for both nominal and numeric attributes). 1Rule can have more than 2 children (for both nominal and numeric) and for numeric attributes have a more complex test than binary split by a value. Also, in WEKA there are 2 different implementations: DecisionStump and OneR. – rapaio Jan 13 '14 at 10:11
• Hmmm...I guess you're right. The original 1R paper says "The specific kind of rules examined in this paper, called 1-Rules, are rules that classify an object on the basis of a single attribute (i.e., they are 1-level decision trees." but decision trees can be implemented in a lot of different ways. I'll edit clear that up. – Matt Krause Jan 13 '14 at 17:42
• There is also a native OneR implementation: The OneR package, on CRAN: CRAN.R-project.org/package=OneR, here is the vignette: cran.r-project.org/web/packages/OneR/vignettes/OneR.html (full disclosure: I am the author of this package). – vonjd Sep 1 '17 at 9:41

Weak learner is a learner that no matter what the distribution over the training data is will always do better than chance, when it tries to label the data. Doing better than chance means we are always going to have an error rate which is less than 1/2.

This means that the learner algorithm is always going to learn something, not always completely accurate i.e., it is weak and poor when it comes to learning the relationships between $X$ (inputs) and $Y$ (target).

But then comes boosting, in which we start by looking over the training data and generate some distributions, then find some set of Weak Learners (classifiers) with low errors, and each learner outputs some hypothesis, $H_x$. This generates some $Y$ (class label), and at the end combines the set of good hypotheses to generate a final hypothesis.

This eventually improves the weak learners and converts them to strong learners.