# bias-variance tradeoff is conflict to bagging

I first give the notations of bias-variance tradeoff:

1. sample set: $$D = \{(x_1,y_1),\cdots,(x_N,y_N)\}.$$

2. relation between $$y$$ and $$x:$$ $$y = f(x) + \epsilon.$$ here $$f(x)$$ is a deterministic function; $$\epsilon \sim (0,\sigma^2)$$ is noise.

3. $$\hat{f}(x;D):$$ the estimation of $$f(x)$$ base on the samples $$D.$$

Then we have the decomposition of mean squared estimation (MSE):

$$E_D[(y-\hat{f}(x;D))^2] =\left(E_D[\hat{f}] - f\right)^2 + E_D[(E_D[\hat{f}] - f)^2] + \sigma^2$$ $$=\left(Bias_D(\hat{f})\right)^2 + Var_D(\hat{f}) + \sigma^2.$$

bias-variance tradeoff usually means that we cannot reduce bias and variance simultaneously. But I confuse that

1. Here we only assume $$y_i$$ is random? i.e. all the randomness is from noise?

2. The tradeoff should be base on the assumption that MSE is fixed ($$\sigma^2$$ is irreducible). Then how do we understand this assumption? Does that mean when the sample set $$D$$ is given, there exists a set of optimal estimations $$\{\hat{f}(x;D)\}$$ minimizing MSE which is the fixed value we mentioned above? And the tradeoff happens among those optimal estimation $$\{\hat{f}(x;D)\}?$$

3. We know that the bagging usually reduces the variance without increasing the bias. Is it conflict to the tradeoff?

Here is my understanding after some discussions:

1. The equation of MSE decomposition is actually nothing about the trade off. It just illustrates that the MSE is from both Bias and Variance.

2. The trade off is a empirical conclusion that 'complex' algorithm usually has low bias but high variance vice versa.

• MSE is not mean squared estimation, it is mean squared error. Regarding 2, we do not assume MSE to be fixed when comparing different models with different amounts of bias and variance. Jan 31, 2021 at 19:36
• @Richard Hardy agree with you. So actually it is wrong way to see the trade off from MSE decompositon in many reference. And is the1. correct？ Feb 1, 2021 at 3:11
• @RichardHardy i mean do we regard $x_i$ as a non-random variable? And the randomness of $\hat{f}$ are all from noise $\epsilon?$ Since in the linear regression, we assume $x$ is non-random. Feb 1, 2021 at 17:35
• Your update makes sense, though the conclusion is both theoretical and empirical. Feb 1, 2021 at 18:21