0
$\begingroup$

I first give the notations of bias-variance tradeoff:

  1. sample set: $D = \{(x_1,y_1),\cdots,(x_N,y_N)\}.$

  2. relation between $y$ and $x:$ $y = f(x) + \epsilon.$ here $f(x)$ is a deterministic function; $\epsilon \sim (0,\sigma^2)$ is noise.

  3. $\hat{f}(x;D):$ the estimation of $f(x)$ base on the samples $D.$

Then we have the decomposition of mean squared estimation (MSE):

$$E_D[(y-\hat{f}(x;D))^2] =\left(E_D[\hat{f}] - f\right)^2 + E_D[(E_D[\hat{f}] - f)^2] + \sigma^2$$ $$=\left(Bias_D(\hat{f})\right)^2 + Var_D(\hat{f}) + \sigma^2.$$

bias-variance tradeoff usually means that we cannot reduce bias and variance simultaneously. But I confuse that

  1. Here we only assume $y_i$ is random? i.e. all the randomness is from noise?

  2. The tradeoff should be base on the assumption that MSE is fixed ($\sigma^2$ is irreducible). Then how do we understand this assumption? Does that mean when the sample set $D$ is given, there exists a set of optimal estimations $\{\hat{f}(x;D)\}$ minimizing MSE which is the fixed value we mentioned above? And the tradeoff happens among those optimal estimation $\{\hat{f}(x;D)\}?$

  3. We know that the bagging usually reduces the variance without increasing the bias. Is it conflict to the tradeoff?

Here is my understanding after some discussions:

  1. The equation of MSE decomposition is actually nothing about the trade off. It just illustrates that the MSE is from both Bias and Variance.

  2. The trade off is a empirical conclusion that 'complex' algorithm usually has low bias but high variance vice versa.

$\endgroup$
4
  • $\begingroup$ MSE is not mean squared estimation, it is mean squared error. Regarding 2, we do not assume MSE to be fixed when comparing different models with different amounts of bias and variance. $\endgroup$ – Richard Hardy Jan 31 at 19:36
  • $\begingroup$ @Richard Hardy agree with you. So actually it is wrong way to see the trade off from MSE decompositon in many reference. And is the1. correct? $\endgroup$ – user6703592 Feb 1 at 3:11
  • $\begingroup$ @RichardHardy i mean do we regard $x_i$ as a non-random variable? And the randomness of $\hat{f}$ are all from noise $\epsilon?$ Since in the linear regression, we assume $x$ is non-random. $\endgroup$ – user6703592 Feb 1 at 17:35
  • $\begingroup$ Your update makes sense, though the conclusion is both theoretical and empirical. $\endgroup$ – Richard Hardy Feb 1 at 18:21
3
$\begingroup$

MSE is not a conserved quantity, like energy in physics. The Bias-Variance trade off is usually a way for us to understand why some models perform better than others.

For a class of models indexed by some complexity parameter, we can conceive as some parameterizations as leading to overly flexible models and some leading to overly inflexible models. The bias variance trade off is then a way to think about balancing those parameterizations. We want to parameterize the models so that their variance drives MSE, but we also want them to be sufficiently flexible so that the bias does not dominate the MSE.

Bagging is an interesting example, and one which I think greatly underscores the importance of understanding the decomposition. Trees are models with low bias but high variance. If we could find a way to reduce that variance we would reduce the MSE (because the bias would be more or less fixed). Just how much bagging reduces the variance can be visualized here. Bagging does not conflict with the tradeoff -- bagging, in my opinion, can be considered a new estimator built from old ones which happens to have smaller variance and similar bias. Hence, smaller error.

$\endgroup$
9
  • $\begingroup$ could I understand as Bias-Variance trade off is actually not a strictly mathematical statement. It just roughly tell you the relation between 'complexity' of model and bias variance? $\endgroup$ – user6703592 Jan 31 at 18:49
  • $\begingroup$ @user6703592 No, because it is a mathematical statement. Changes in the complexity parameter do not result in giving units of error from the bias term to the variance term of vice versa. All the trade off is saying is that the error is the sum of these two quantities plus an irreducible error term. Our job is to find complexity parameters which make this error smallest. When we change the parameter, we can think of that change as resulting in more or less error, and where that error comes from is either from bias or variance, depending on if we make the model more or less complex. $\endgroup$ – Demetri Pananos Jan 31 at 19:07
  • $\begingroup$ I think of this mathematical statement as the bias-variance decomposition. The tradeoff is an empirical observation, that often increasing complexity trades away bias for variance. And then the goal is to minimize total MSE by finding the optimal tradeoff. $\endgroup$ – Ben Reiniger Jan 31 at 19:14
  • $\begingroup$ @BenReiniger Yea, I like that. $\endgroup$ – Demetri Pananos Jan 31 at 19:19
  • $\begingroup$ agree with you. So actually it is wrong way to see the trade off from MSE decompositon in many reference. And is the 1. correct? $\endgroup$ – user6703592 Feb 1 at 3:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.