Apparently, a learning algorithm must make a trade off between bias and variance when producing a hypothesis. Bias means systematic deviation from data. Variance refers to the error due to fluctuations when applying the hypothesis to different training sets.
Why must there be a trade off here?
It results as a decomposition of the error function in two terms, representing "two opposing forces", in the sense that in order to reduce the bias error, you need your model to consider more possibilities to fit the data. But this on the other side increases the variance error. Also, the other way around: if your model fits too much (starts to fit noise, which you could see as non-systematic variations on your individual samples), then you need to force your parameters not to vary too wildly, and thus introducing bias.
In more intuitive terms: bias error is being systematically wrong, and variance error is about learning all tiny, accidental variations of the samples.
Take a look at this nice article for details, http://scott.fortmann-roe.com/docs/BiasVariance.html
There are some good answers here. Let me fill a small gap by providing a simple answer to the question as stated: "Why must there be a tradeoff between bias and variance?"
The answer is fairly straightforward. Estimates of population parameters based on sample data will vary from sample to sample, because the sample data will vary. This means that there will be a distribution of parameter estimates under repeated sampling. To understand the pros and cons of different algorithms for estimating population parameters, we need to know about the properties of their sampling distributions. Specifically, we want to know if the sampling distribution is centered on the true value (i.e., if it's biased), and we want to know how 'wide' the sampling distribution is (i.e., how far away from the true value the estimates tend to be on average--this is the variance part).
Now it happens that there will be an estimation procedure that will have the smallest variance of all possible estimation procedures that are unbiased, but there will also often be another estimation procedure that will have even smaller variance (but which will be biased). In this situation, the analyst is faced with a decision regarding which procedure they want to use, depending on whether unbiasedness or lower variance is more important to them in that situation.
In sum: there must be a tradeoff between bias and variance anytime there is more than one possible estimation procedure where the one with the minimum variance of all unbiased procedures is not the procedure with the minimum variance overall.
In general, this is about finding a compromise between a) cases where Hypothesis NULL is true, but is rejected and b) cases where Hypothesis NULL is wrong, but fails to be rejected.
This is a very vague topic, but restricting to Hypothesis-testing, I suggest going through: Type I and Type II errors involved.
And also, if possible do mention what learning algorithms you are referring to. Thanks!
A bias / variance trade off exists anytime a non-parametric model is specified. A machine learning algorithm might be used to determine the bin width for a non-parametric model. To see why there is a bias-variance trade off, think about a toy example where we model weight as a function of calories.
Imagine that your independent variable, calories consumed, is continuously valued and ranges from 2,000 to 4,000 calories.
If we had to guess the average weight for a particular individual drawn at random from the entire sample, our best guess would be the average weight across all levels of calorie consumption in our sample. This guess may not be the most accurate guess we can come up with, because it doesn't take into account any individual characteristics. Although the guess may not be the most accurate (may be biased), it also has a low variance. It may be biased because our prediction for weight is the same regardless of whether we are told that an individual consumes 0 or 1,000,000 calories in a day. The variance is low because adding or removing an observation doesn't change our average weight of the sample by much if our sample size is large.
If we consider calorie consumption, our guess gets a little bit better. Consider running a regression where we model weight as a function of an intercept term and a term describing the daily caloric intake of an individual. The intercept tells us the average weight within the sample. The coefficient on "calories" tells us that our best guess for weight will change by "Beta" for every additional calorie consumed in a day.
But, non-parametric regression goes one step further. Consider this idea: the marginal effect of an additional calorie depends on how many calories the individual usually consumes in a day. Consider two cases - an emaciated adolescent compared with an Olympic Athlete. Perhaps the adolescent is only consuming 1,500 calories in a day. Because this amount is deficient, the marginal effect of an additional calorie is large. Consuming more food in a day for the child means gaining weight (in a possibly healthy way). In the case of the Olympic Athlete, perhaps they burn everything they consume due to rigorous training schedules. In this case, we may consider that consuming an additional calorie has little effect on weight.
So, perhaps increased caloric intake has a diminishing marginal effect on an individual's weight.
If we were to incorporate this into our model, we may create "bins" which partition our independent variable, "daily caloric intake" in a way that better explains our dependent variable, "weight". Usually when creating non-parametric models, we might assume that our data are uniformly and evenly distributed within each bin.
Start by incorporating two bins into our model: one for individuals who are "calorie deficient" (below 2,500 calories / day) and one for individuals who are "calorie sufficient" (above 2,500 calories / day). Regressing weight as a function of caloric intake, as well as indicator variables for "calorie deficient" and "calorie sufficient" may tell us a better story. We may realize there is a new relationship between calories and weight, and that our "Beta" coefficient on "calories" has now changed because we removed some bias. But we also have new marginal effects on top of that. The coefficients on our indicator variables describe how the marginal effect of an additional calorie changes depending on how many calories the individual usually consumes.
By creating bins, we are forcing our model to "predict the average weight within each bin". As the width of the bins decreases (as our bins get smaller and include a smaller range of caloric intake), the average weight in each bin converges to the observed weights in each bin, because the average is based off of fewer data points with a smaller range. However, the smaller number of observations means that this average comes with a higher variance. Changing one observation has a large effect on our average guess within each bin.
At the extreme, we can create bins with only one observation. The average of this bin will be equivalent to the observed level of caloric intake, which implies that our estimator has 0 bias. However, the variance is very large because our sample within each bin is very small (size of 1), and our average is very susceptible to changes in our sample. If we randomly resampled from our population, our estimator for a particular bin would surely be different. Our estimator has lots of variance, but is not biased.
As the width of bins increases (as our bins get larger and include a larger range of caloric intake), the average weight in each bin may have additional bias. We are forcing the marginal effect of a calorie to be the same within a larger group of individuals, when in reality this may not be the case. However, adding or removing an observation within this bin has very little effect on our estimator (the variance of our estimator is very low).
So, in general: non-parametric regressions suffer from a bias-variance trade off. Partitioning data into smaller bins means that our estimators for each bin will have less bias at the cost of our estimators having a high variance, whereas partitioning data into larger bins means that our estimators for each bin will have less variance at the cost of having more bias. Consider the attributes of your average "guess" within each bin and how they will change as the size of the bins changes: 1 data point or observation in a bin implies a "perfect" average which is very susceptible to changes in data (large variance), versus an infinite number of points in a bin which implies an imperfect average (the average doesn't perfectly guess every point in the bin) with very little variance (adding / dropping an observation has little to no effect).
