Bias / variance tradeoff math

Question

I understand the matter in the underfitting / overfitting terms but I still struggle to grasp the exact math behind it. I've checked several sources (here, here, here, here and here) but I still don't see why exactly bias and variance oppose each other like, e.g., $e^x$ and $e^{-x}$ do:

source

It seems like everybody derives the following equation (omitting the irreducible error $\epsilon$ here) $$\newcommand{\var}{{\rm Var}} E[(\hat{\theta}_n - \theta)^2]=E[(\hat{\theta}_n - E[\hat{\theta}_n])^2] + (E[\hat{\theta}_n - \theta])^2 $$ and then, instead of driving the point home and showing exactly why the terms on the right behave the way they do, starts wandering about the imperfections of this world and how impossible it is to be both precise and universal at the same time.

The obvious counterexample

Say, a population mean $\mu$ is being estimated using sample mean $\bar{X}_n = \frac{1}{n}\sum\limits_{i=1}^{n}X_i$, i.e. $\theta\equiv\mu$ and $\hat{\theta}_n\equiv\bar{X}_n$ then: $$MSE = \var(\bar{X}_n - \mu) + (E[\bar{X}_n] - \mu)^2 $$ since $E[\bar{X}_n]=\mu$ and $\var(\mu) = 0$, we have: $$MSE = \var(\bar{X}_n) = \frac{1}{n}\var(X)\xrightarrow[n\to\infty]{}0$$

So, the questions are:

1. Why exactly $E[(\hat{\theta}_n - E[\hat{\theta}_n])^2]$ and $E[\hat{\theta}_n - \theta]$ cannot be decreased simultaneously?
2. Why can't we just take some unbiased estimator and reduce the variance by increasing sample size?

Answer 1

First, nobody says that squared bias and variance behave just like $e^{\pm x}$, in case you are wondering. The point simply is that one increases and the other decreases. It'd similar to supply and demand curves in microeconomics, which are traditionally depicted as straight lines, which sometimes confuses people. Again, the point simply is that one slopes downward and the other upward.

Your key confusion is about what is on the horizontal axis. It's model complexity - not sample size. Yes, as you write, if we use some unbiased estimator, then increasing the sample size will reduce its variance, and we will get a better model. However, the bias-variance tradeoff is in the context of a fixed sample size, and what we vary is the model complexity, e.g., by adding predictors.

If model A is too small and does not contain predictors whose true parameter value is nonzero, and model B encompasses model A but contains all predictors whose parameter values are nonzero, then parameter estimates from model A will be biased and from model B unbiased - but the variance of parameter estimates in model A will be smaller than for the same parameters in model B.

Answer 2

Problems occur when a model $f(x,\theta)$ has a high tendency to fit the noise.

In that case the model tends to over-fit. That is, it is not only expressing the true model but also the random noise that you do not want to capture with your model (because the noise is a non-systematic part that does not allow you to make predictions for new data).

One might improve (reduce) the total error of fitting, by introducing some bias, when the this bias makes the variance/over-fitting reduce more strongly than the increase of the bias/under-fitting (ie not correctly representing the true model).

1. Why exactly $E[(\hat{\theta}_n - E[\hat{\theta}_n])^2]$ and $E[\hat{\theta}_n - \theta]$ cannot be decreased simultaneously?

This is not true. They can be decreased simultaneously (depending on the case). Imagine that you introduced some bias which both increased the variance as well as the bias. Then in the reverse direction reducing this bias will simultaneously reduce bias and variance.

^{For example a scaled root mean squared difference $c \sqrt{\frac{1}{n} {\sum(x_i-\bar{x})^2}}$ for sample of size $n$ is an unbiased estimator for the population standard deviation $\sigma$ when $c=\sqrt{\frac{n}{n-1}}$. Now, if you would have $c>\sqrt{\frac{n}{n-1}}$, then you would both reduce the bias as well as the variance when you reduce the size of this constant $c$.}

However, the bias that is (intentionally) added in regularization is often of the kind that reduces the variance (e.g. you could reduce $c$ to a level below $\sqrt{\frac{n}{n-1}}$). Thus, in practice, you get a trade-off in bias versus variance and reducing the bias will increase the variance (and vice versa).

2. Why can't we just take some unbiased estimator and reduce the variance by increasing sample size?

In principle you can.

But,

• This may require much more sampling effort which is expensive, and this is often a limitation.
• Possibly there might also be computational difficulties with certain estimation problems and the sample size would need to increase extremely in order to solve this, if it is possible at all.

(e.g. high dimensionality parameters>measurements, or as in ridge regression: very shallow paths around the global optimum)

Often there is also no objection to bias. When it is about reducing the total error (as in many cases) then the use of a biased but less erroneous estimator is to be preferred.

About your counter example.

Related to your second question you can indeed reduce the error by increasing the sample size. And related to your first question you can also reduce both bias and variance (say you use a scaled sample mean $c\frac{\sum{x_i}}{n}$ as estimator of the population mean and consider varying the scaling parameter $c$).

However the region of practical interest is where the decreasing bias coincides with an increasing variance. The image below shows this contrast by using a sample (size = 5) taken from a normal distribution with variance = 1 and mean = 1. The unscaled sample mean is the unbiased predictor of the population mean. If you would increase the scaling of this predictor than you have both increasing bias and increasing variance. However if you decrease the scaling of the predictor then you have increasing bias, but decreasing variance. The "optimal" predictor is then actually not the sample mean but some shrinked estimator (see also Why is the James-Stein estimator called a "shrinkage" estimator?).