bias-variance tradeoff is conflict to bagging

Question

I first give the notations of bias-variance tradeoff:

1. sample set: $D = \{(x_1,y_1),\cdots,(x_N,y_N)\}.$

2. relation between $y$ and $x:$ $y = f(x) + \epsilon.$ here $f(x)$ is a deterministic function; $\epsilon \sim (0,\sigma^2)$ is noise.

3. $\hat{f}(x;D):$ the estimation of $f(x)$ base on the samples $D.$

Then we have the decomposition of mean squared estimation (MSE):

$$E_D[(y-\hat{f}(x;D))^2] =\left(E_D[\hat{f}] - f\right)^2 + E_D[(E_D[\hat{f}] - f)^2] + \sigma^2$$ $$=\left(Bias_D(\hat{f})\right)^2 + Var_D(\hat{f}) + \sigma^2.$$

bias-variance tradeoff usually means that we cannot reduce bias and variance simultaneously. But I confuse that

1. Here we only assume $y_i$ is random? i.e. all the randomness is from noise?

2. The tradeoff should be base on the assumption that MSE is fixed ($\sigma^2$ is irreducible). Then how do we understand this assumption? Does that mean when the sample set $D$ is given, there exists a set of optimal estimations $\{\hat{f}(x;D)\}$ minimizing MSE which is the fixed value we mentioned above? And the tradeoff happens among those optimal estimation $\{\hat{f}(x;D)\}?$

3. We know that the bagging usually reduces the variance without increasing the bias. Is it conflict to the tradeoff?

Here is my understanding after some discussions:

1. The equation of MSE decomposition is actually nothing about the trade off. It just illustrates that the MSE is from both Bias and Variance.

2. The trade off is a empirical conclusion that 'complex' algorithm usually has low bias but high variance vice versa.

Answer 1

MSE is not a conserved quantity, like energy in physics. The Bias-Variance trade off is usually a way for us to understand why some models perform better than others.

For a class of models indexed by some complexity parameter, we can conceive as some parameterizations as leading to overly flexible models and some leading to overly inflexible models. The bias variance trade off is then a way to think about balancing those parameterizations. We want to parameterize the models so that their variance drives MSE, but we also want them to be sufficiently flexible so that the bias does not dominate the MSE.

Bagging is an interesting example, and one which I think greatly underscores the importance of understanding the decomposition. Trees are models with low bias but high variance. If we could find a way to reduce that variance we would reduce the MSE (because the bias would be more or less fixed). Just how much bagging reduces the variance can be visualized here. Bagging does not conflict with the tradeoff -- bagging, in my opinion, can be considered a new estimator built from old ones which happens to have smaller variance and similar bias. Hence, smaller error.