Bias-variance: is it really a "trade-off"?

Question

For some estimator $\hat{\theta}$, we have the "bias-variance trade-off": $MSE(\hat{\theta}) = bias^2(\hat{\theta}) + var(\hat{\theta}).$

When I think of a trade-off, I would expect that as the variance goes down, the squared bias would go up, and vice versa. But I'm not sure this is the case here, since $MSE(\hat{\theta})$ is not fixed as $\hat{\theta}$ as varies.

Example: Consider two silly estimates for the mean of a single normally distributed data point $x \sim N(\theta, 1)$: $\delta_0(x) = 0$ and $\delta_1(x) = 1$. Both of these has the same variance of $0$. Furthermore, $bias^2(\delta_0) = \theta^2$ and $bias^2(\delta_1) = (1-\theta)^2$. Therefore, both estimators have the same bias but different squared biases (assuming $\theta \ne \frac{1}{2}$).

So in what sense is there a trade-off here?

Answer 1

I share your skepticism that there is a tradeoff. A typical way to think about the bias-variance decomposition of MSE, such as in regularized regression, is that we accept a bit of bias in our estimator in exchange for a large reduction in variance. However, we do this to achieve lower MSE, not to maintain the MSE. Thus, while I understand the "trade" being used to describe trading your unbiased, high variance estimator for a slightly biased, low variance estimator, "tradeoff" to me implies keeping MSE constant, and I try not to describe it as a "tradeoff", preferring to refer to a bias-variance "decomposition".

Answer 2

Here is an example: suppose $X_1,X_2,\cdots, X_n$ are iid samples from $\mathcal N(\mu,\sigma^2)$, where you do not know $\mu,\sigma^2$ and you want to estimate them. The natural estimator of $\mu$ is $\hat \mu=\bar X= \frac1n \sum X_i$.

It is less immediately obvious what to use to estimate $\sigma^2$ but you might expect to use some fraction of $\sum(X_i-\bar X)^2$, which has mean $(n-1)\sigma^2$ and variance $2(n-1)\sigma^4$, so $\frac1k\sum(X_i-\bar X)^2$ has mean $\frac{n-1}{k}\sigma^2$ and variance $\frac{2(n-1)}{k^2}\sigma^4$. Not using a fraction of $\sum(X_i-\bar X)^2$ might lead to some of the inefficiencies in the "silly estimates" in your question.

This means that (the square of) the bias increases as $k$ moves away from $n-1$, while the variance reduces as $k$ increases. Reducing $k$ below $n-1$ tends to increase both so it is presumably unwise, but increasing $k$ above $n-1$ leads to a tradeoff with the square of the bias increasing and the variance reducing.

Here is an chart, with the square of the bias in red, the variance in blue and the expected mean-square-error in black (it uses $n=6$ but this makes little difference). This illustrates:

• the unbiased estimator $\frac1{n-1}\sum(X_i-\bar X)^2$ has expected mean-square-error of $\frac{2}{n-1}\sigma^4$; lower values of $k$ perform worse

• the maximum likelihood estimator $\frac1{n}\sum(X_i-\bar X)^2$ has expected mean-square-error of $\frac{2n-1}{n^2}\sigma^4$, which is smaller despite a biased estimator

• expected mean-square-error is minimised with $k=n+1$ i.e. with $\frac1{n+1}\sum(X_i-\bar X)^2$ when it is $\frac{2}{n+1}\sigma^4$, while for higher values of $k$ the smaller variance fails to offset the larger square of the bias.

Answer 3

The "trade off" relates to the situations where we control overfitting and underfitting. Situations that are prone to overfitting can be managed by introducing bias which reduces the tendency to fit the random noise component instead of the deterministic part of the model.

^[source]

It is not a general rule that bias and variance are always a tradeoff. It is possible that both variance and bias increase. See also: Bias / variance tradeoff math

In the image below from that question you see the effect of shrinkage on the bias and variance of an estimator.

• There is a way to introduce bias while reducing variance, and in such a case there is really a tradeoff.
• That doesn't mean that every bias is a tradeoff with variance, and there are also ways to introduce bias while increasing variance.