Going through this lecture note on bias-variance trade-off, I didn't follow the latter part of this paragraph.

It shows the common situation in practice that

(1) for simple models, the bias increases very quickly, while

(2) for complex models, the variance increases very quickly. Since the riskiness is additive in the two, the optimal complexity is somewhere in the middle. Note, however, that these properties do not follow from the bias-variance decomposition, and need not even be true.

The 'It' in the above paragraph refers to the below image:

enter image description here


1) If these are properties then why don't they follow from the bias variance decomposition, which states that: $ E[(Y-\hat{f}(x))^2] = \sigma^2 + \text{Bias}^2 + \text{Var}(\hat{f}(x))$.

2) And, under what conditions are they not true?

  • $\begingroup$ 1) why should they follow from the decomposition ? It doesn't say anything about complexity. The definition you have in 1) is for a given model complexity. As far 2), if you somehow had the true underlying model. ( not estimated ), then, because there is no concept of complexity in this case ( we know the true model), then the bias won't be a function of complexity and neither would the variance , so the relation wouldn't be true in that case. $\endgroup$ – mlofton Feb 1 '19 at 15:18
  • 1
    $\begingroup$ There may be a potential to mis-read the quotation. "These properties" refers to the assertions about rates of increase in bias and variance for "simple" and "complex" models, not to the bias and variance. At best these properties are heuristics--and obviously they are separate from the bias-variance trade-off relationship. $\endgroup$ – whuber Feb 1 '19 at 20:30
  • $\begingroup$ @mloftonThe definition in 1 is for any model complexity. Re 2: why is there no complexity if we have the true model. Your second point contradicts the first. You say in first that the decomposition does not say anything about complexity and at the same time in second you implicitly assume that it does. $\endgroup$ – naive Feb 1 '19 at 20:32
  • $\begingroup$ @whuber- I must agree to your comment that these properties are heuristics. But they are nevertheless important to understand the bias variance trade-off relationships in a very intuitive way. I am just looking for something that will supplement my understanding. Cheers! $\endgroup$ – naive Feb 1 '19 at 20:38
  • 2
    $\begingroup$ While @whuber is correct that these are heuristics, another way to think about things is as a guide to what we mean by model complexity. Any reasonable definition of model complexity will cause these properties to be true. $\endgroup$ – Matthew Drury Feb 1 '19 at 20:50

As mentioned by @whuber in comments, these properties are heuristics and they talk about the rates of increase in bias and variance and not the bias and variance with respect to the model complexity.

From Wikipedia:

In statistics and machine learning, the bias–variance tradeoff is the property of a set of predictive models whereby models with a lower bias in parameter estimation have a higher variance of the parameter estimates across samples, and vice versa. 

And on bias-variance decomposition:

The bias–variance decomposition is a way of analyzing a learning algorithm's expected generalization error with respect to a particular problem as a sum of three terms, the bias, variance, and a quantity called the irreducible error, resulting from noise in the problem itself.

So, the property that has been referred to in the question --bias variance tradeoff -- is a property of a set of predictive models, in general. As noted by @whuber the image in the question is a way of demonstrating that property heuristically.

And, the bias-variance decomposition is a way to analyze the generalization error of an algorithm for a particular problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.