7
$\begingroup$

Let us consider the bias-variance decomposition in the context of model selection. The picture below suggests the optimal model (the one minimizing the expected squared prediction error) will have $\text{Variance}=\text{Bias}^2$.

enter image description here

This looks like a very special case, as corresponding diagrams in machine learning textbooks are often asymmetric. See e.g. James et al. "Introduction to Statistical Learning" Figure 2.12:

enter image description here

I think the curious result of the initial picture rests on the curves of variance and squared bias being convex and rather symmetric. The convexity is probably sensible, but I am not so sure about the approximate symmetry.

Question: What are some concrete settings (model classes and data generating processes) in which $\text{Variance}=\text{Bias}^2$ could be expected to hold (at least approximately) for the optimal model?

P.S. There is no need to address the question of whether the initial picture must hold in general, as that may lead this thread off track. The answer is clearly negative, as indicated above and in Stephen Kolassa's answer.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

The picture below suggests the optimal model (the one minimizing the expected squared prediction error) will have $\text{Variance}=\text{Bias}^2$.

Sorry to rain on your parade, but this does not necessarily hold. The picture is misleading. No, that is not an answer to your question, but I would assume this reduces our interest in it altogether...

As an example, assume model complexity $x$ can be parameterized in a single dimension with $0\leq x\leq 1$. Squared bias is given by $$ b(x) = 1-\sqrt{x}, $$ variance by $$v(x) = x^2, $$ and total error therefore by $$ e(x) = b(x)+v(x) = 1-\sqrt{x}+x^2. $$

bias and variance

Bias is decreasing, variance is increasing, both are convex, and the minimal error is achieved at $x=\frac{1}{4^\frac{2}{3}}$, far away from the point of intersection.

The result about minimal error appearing at the intersection should indeed hold if we posit symmetry of bias and variance. But then they would need to be symmetric about some specific $x$ value, and that particular $x$ value would then turn out to be the minimum error complexity. I would not think such a symmetry to be very common, or therefore interesting.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Well, the parade is actually not mine. I was pretty surprised by the picture (I am much more used to ones like yours), but I found it curious, so I shared it here. Your last paragraph is getting at my actual question and contains a hint at an answer. It would be nice if you or someone else could expand on it more. Note also symmetry is not enough; convexity is also needed. $\endgroup$
    – Richard Hardy
    Commented Mar 18, 2021 at 11:13
  • $\begingroup$ In the error $e(x)$ the bias is not squared, It must be. $\endgroup$
    – markowitz
    Commented Mar 18, 2021 at 14:55
  • $\begingroup$ @markowitz: $b$ was supposed to refer to squared bias, I have edited the post $\endgroup$
    – Stephan Kolassa
    Commented Mar 18, 2021 at 14:58
  • $\begingroup$ @StephanKolassa; Ok. But I have a more relevant point. Your is just an example but in the same fashion we can build another example where the “symmetry” above hold. I agree with your conclusion “I would not think such a symmetry to be very common, or therefore interesting.” (see my answer) but it seems me that the major motivations do not appear from your explanation. $\endgroup$
    – markowitz
    Commented Mar 18, 2021 at 15:40
  • $\begingroup$ @markowitz: fair enough. Yes, of course, this is just an example. It's just that the "general" relationship that is suggested in the symmetrical picture does not hold. As you write, a general rule will not hold, and a specific example where this symmetry actually does hold will be a very contrived one. I see the question has been edited. Note that my answer was posted in response to the original one. $\endgroup$
    – Stephan Kolassa
    Commented Mar 18, 2021 at 15:43
4
$\begingroup$

When will $Variance=Bias^2$ hold for the optimal model?

I think that some ad hoc example can be built but a general rule cannot exist.

First of all is useful to say that the bias-variance tradeoffs (BVT) story matters in prediction only. Read here (What is the relationship between minimizing prediciton error versus parameter estimation error?). So, we have a true model in one side and several "proposed/estimated" models in another; among the last we looking for the best one.

In general, the level of complexity in the proposed/estimated model that minimize the MSE, depends crucially on the true model. Then, It can seems that BVT story can suggest us that, under a true model with mid level of complexity, something like $Variance = Bias^2$ can hold for the best estimated model (min MSE).

However this perspective do not take into account that the BVT must be analyzed under a precise amount of data. The best proposed/estimated model change if the number of available data change; indeed if we have an infinite amount of data only $Bias$ matters.

Therefore, the relevance of the “symmetry” linked to $Variance = Bias^2$ decrease substantially.

This topic is related Statistical Learning. Contradictions?

Improve this answer
$\endgroup$
6
  • $\begingroup$ True, BVT must be analyzed under a fixed amount of data. But that is often the case; we have the data that we have and then we are looking for a good predictive model. $\endgroup$
    – Richard Hardy
    Commented Mar 18, 2021 at 14:41
  • $\begingroup$ Sure. But, as said above, this fact decrease substantially the relevance of the question. Sophisticated reasonings around the true model become much less important. Agree? I think that the graph above represent only a snapshot for figured out the BVT message; not so realistic or geometrically important. $\endgroup$
    – markowitz
    Commented Mar 18, 2021 at 15:04
  • $\begingroup$ Hmm, I am not sure I get the point. The relevance does not decrease if the situation is very common. And I think it is, because we usually have a fixed data sample that we want to analyze. Or at least I do. There can be occasions where one can collect more data, but that is not always possible and often costly. And the snapshot is unfortunately misleading as it implies Variance = Bias^2 which is rarely the case. It would be nice to find out how commonly it may apply, hence the question. $\endgroup$
    – Richard Hardy
    Commented Mar 18, 2021 at 15:13
  • $\begingroup$ But what situation is very common? You have some $n$ but $n=10, 100, 300, 2000, … ?$ … if the symmetry you are interested in depend, among other things, from $n$ … it seems me impossible that it is a common or relevant situation. This is my point. If it was true that this symmetry depend only about the true model … become more interesting what kind of true model imply it. But it is not so; from dimensionality problem, no kind of true model can imply the summetry you are interested in. $\endgroup$
    – markowitz
    Commented Mar 18, 2021 at 15:32
  • $\begingroup$ Hm. What is "variance" and "bias" if "we have the data that we have"? What do we take expectations over? I'd say we need to consider the BVT in the context of a specific amount of data, not a specific set of observations. $\endgroup$
    – Stephan Kolassa
    Commented Mar 18, 2021 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.