I understand the matter in the underfitting / overfitting terms but I still struggle to grasp the exact math behind it. I've checked several sources (here, here, here, here and here) but I still don't see why exactly bias and variance oppose each other like, e.g., $e^x$ and $e^{-x}$ do:


It seems like everybody derives the following equation (omitting the irreducible error $\epsilon$ here) $$\newcommand{\var}{{\rm Var}} E[(\hat{\theta}_n - \theta)^2]=E[(\hat{\theta}_n - E[\hat{\theta}_n])^2] + (E[\hat{\theta}_n - \theta])^2 $$ and then, instead of driving the point home and showing exactly why the terms on the right behave the way they do, starts wandering about the imperfections of this world and how impossible it is to be both precise and universal at the same time.

The obvious counterexample

Say, a population mean $\mu$ is being estimated using sample mean $\bar{X}_n = \frac{1}{n}\sum\limits_{i=1}^{n}X_i$, i.e. $\theta\equiv\mu$ and $\hat{\theta}_n\equiv\bar{X}_n$ then: $$MSE = \var(\bar{X}_n - \mu) + (E[\bar{X}_n] - \mu)^2 $$ since $E[\bar{X}_n]=\mu$ and $\var(\mu) = 0$, we have: $$MSE = \var(\bar{X}_n) = \frac{1}{n}\var(X)\xrightarrow[n\to\infty]{}0$$

So, the questions are:

  1. Why exactly $E[(\hat{\theta}_n - E[\hat{\theta}_n])^2]$ and $E[\hat{\theta}_n - \theta]$ cannot be decreased simultaneously?
  2. Why can't we just take some unbiased estimator and reduce the variance by increasing sample size?

First, nobody says that squared bias and variance behave just like $e^{\pm x}$, in case you are wondering. The point simply is that one increases and the other decreases. It'd similar to supply and demand curves in microeconomics, which are traditionally depicted as straight lines, which sometimes confuses people. Again, the point simply is that one slopes downward and the other upward.

Your key confusion is about what is on the horizontal axis. It's model complexity - not sample size. Yes, as you write, if we use some unbiased estimator, then increasing the sample size will reduce its variance, and we will get a better model. However, the bias-variance tradeoff is in the context of a fixed sample size, and what we vary is the model complexity, e.g., by adding predictors.

If model A is too small and does not contain predictors whose true parameter value is nonzero, and model B encompasses model A but contains all predictors whose parameter values are nonzero, then parameter estimates from model A will be biased and from model B unbiased - but the variance of parameter estimates in model A will be smaller than for the same parameters in model B.

  • 2
    $\begingroup$ Thank you for the answer. I mentioned $e^x$ solely to illustrate the point of obviously opposing functions. Either way, are you saying that the tradeoff is an attribute of multivariate systems and cannot be easily shown in the univariate case? Qualitatively speaking I get the point of complexity of the model vs overfitting but can it be shown mathematically? $\endgroup$ – ayorgo Mar 24 '18 at 11:49
  • 2
    $\begingroup$ You can show it mathematically if you restrict yourself to a specific model class, e.g., Ordinary Least Squares. In the simplest case, the true DGP may depend linearly on a single variable $x$. Model A would then be a simple mean model, and model B would be a regression on $x$, and you can calculate bias and variance. And if you want to, you can include higher powers of $x$ for even more variance. $\endgroup$ – Stephan Kolassa Mar 24 '18 at 12:14
  • $\begingroup$ The values the OP mentions are population values. The estimates of these values may have non-zero correlation, e.g. King and Zhen: gking.harvard.edu/files/gking/files/0s.pdf see page 11 where they state "and so we are in the happy situation where reducing bias also reduces variance". However, as Stephan mentions the horizontal axis of the plot in the OP is model complexity and the example given by King and Zheng, is by default more complex than a logistic regression. $\endgroup$ – Lucas Roberts Mar 24 '18 at 14:06

Problems occur when a model $f(x,\theta)$ has a high tendency to fit the noise.

In that case the model tends to over-fit. That is, it is not only expressing the true model but also the random noise that you do not want to capture with your model (because the noise is a non-systematic part that does not allow you to make predictions for new data).

One might improve (reduce) the total error of fitting, by introducing some bias, when the this bias makes the variance/over-fitting reduce more strongly than the increase of the bias/under-fitting (ie not correctly representing the true model).

1. Why exactly $E[(\hat{\theta}_n - E[\hat{\theta}_n])^2]$ and $E[\hat{\theta}_n - \theta]$ cannot be decreased simultaneously?

This is not true. They can be decreased simultaneously (depending on the case). Imagine that you introduced some bias which both increased the variance as well as the bias. Then in the reverse direction reducing this bias will simultaneously reduce bias and variance.

For example a scaled root mean squared difference $c \sqrt{\frac{1}{n} {\sum(x_i-\bar{x})^2}}$ for sample of size $n$ is an unbiased estimator for the population standard deviation $\sigma$ when $c=\sqrt{\frac{n}{n-1}}$. Now, if you would have $c>\sqrt{\frac{n}{n-1}}$, then you would both reduce the bias as well as the variance when you reduce the size of this constant $c$.

However, the bias that is (intentionally) added in regularization is often of the kind that reduces the variance (e.g. you could reduce $c$ to a level below $\sqrt{\frac{n}{n-1}}$). Thus you get a trade-off in bias versus variance and removing the bias will (in practice) increase the variance.

2. Why can't we just take some unbiased estimator and reduce the variance by increasing sample size?

In principle you can.


  • This may require much more sampling effort which is expensive, and this is often a limitation.
  • Possibly there might also be computational difficulties with certain estimation problems and the sample size would need to increase extremely in order to solve this, if it is possible at all.

    (e.g. high dimensionality parameters>measurements, or as in ridge regression: very shallow paths around the global optimum)

Often there is also no objection to bias. When it is about reducing the total error (as in many cases) then the use of a biased but less erroneous estimator is to be preferred.

About your counter example.

Related to your second question you can indeed reduce the error by increasing the sample size. And related to your first question you can also reduce both bias and variance (say you use a scaled sample mean $c\frac{\sum{x_i}}{n}$ as estimator of the population mean and consider varying the scaling parameter $c$).

However the region of practical interest is where the decreasing bias coincides with an increasing variance. The image below shows this contrast by using a sample (size = 5) taken from a normal distribution with variance = 1 and mean = 1. The unscaled sample mean is the unbiased predictor of the population mean. If you would increase the scaling of this predictor than you have both increasing bias and increasing variance. However if you decrease the scaling of the predictor then you have increasing bias, but decreasing variance. The "optimal" predictor is then actually not the sample mean but some shrinked estimator (see also Why is the James-Stein estimator called a "shrinkage" estimator?).

overfitting and underfitting in shrinking of sample mean


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.