Bias-variance decomposition of squared error

Question

In the equation for expected test error we are summing the function's bias, variance and noise.

I am not quite sure why we are also summing the variance of the function. My intuition says that more flexible functions, which have higher variance, can copy the error of our training data too much which will cause overfitting. But we can also have good data with little noise, so why would choosing a more flexible function automatically increase the expected test error?

Or in other words, Why are we also summing the variance in that equation?

Answer 1

This is the Bias-Variance decomposition of square loss. To make things a bit concrete let:

$\theta$ - be a paramater we are trying to estimate

$\hat{\theta}$ - be an estimator of $\theta$

So for example, if you were trying to estimate the mean of $N(\mu,1)$, the parameter we are trying to estimate, $\theta$, is $\mu$ and a common way to estimate is by computing the sample mean $\sum_{i=1}^n \frac{X_i}{n}$, this is $\hat{\theta}$ in this case. So estimator is the function you use to try and estimate the parameter.

Now estimators are usually evaluated by using square norm, this is: $$E (\theta - \hat{\theta})^2 $$ The smaller this value is, we say the better your estimator is performing.

Now, there is an equivalent way to compute the square norm, that is: $$E(\theta - \hat{\theta})^2 = Var(\hat{\theta}) + Bias(\hat{\theta})^2 $$ That's all thats going on. If you are estimating functions replace the function you are trying to estimate $f$ with its estimator $\hat{f}$