12
$\begingroup$

I understand the matter in the underfitting / overfitting terms but I still struggle to grasp the exact math behind it. I've checked several sources (here, here, here, here and here) but I still don't see why exactly bias and variance oppose each other like, e.g., $e^x$ and $e^{-x}$ do:


source

It seems like everybody derives the following equation (omitting the irreducible error $\epsilon$ here) $$\newcommand{\var}{{\rm Var}} E[(\hat{\theta}_n - \theta)^2]=E[(\hat{\theta}_n - E[\hat{\theta}_n])^2] + (E[\hat{\theta}_n - \theta])^2 $$ and then, instead of driving the point home and showing exactly why the terms on the right behave the way they do, starts wandering about the imperfections of this world and how impossible it is to be both precise and universal at the same time.

The obvious counterexample

Say, a population mean $\mu$ is being estimated using sample mean $\bar{X}_n = \frac{1}{n}\sum\limits_{i=1}^{n}X_i$, i.e. $\theta\equiv\mu$ and $\hat{\theta}_n\equiv\bar{X}_n$ then: $$MSE = \var(\bar{X}_n - \mu) + (E[\bar{X}_n] - \mu)^2 $$ since $E[\bar{X}_n]=\mu$ and $\var(\mu) = 0$, we have: $$MSE = \var(\bar{X}_n) = \frac{1}{n}\var(X)\xrightarrow[n\to\infty]{}0$$

So, the questions are:

  1. Why exactly $E[(\hat{\theta}_n - E[\hat{\theta}_n])^2]$ and $E[\hat{\theta}_n - \theta]$ cannot be decreased simultaneously?
  2. Why can't we just take some unbiased estimator and reduce the variance by increasing sample size?
$\endgroup$

2 Answers 2

12
$\begingroup$

First, nobody says that squared bias and variance behave just like $e^{\pm x}$, in case you are wondering. The point simply is that one increases and the other decreases. It'd similar to supply and demand curves in microeconomics, which are traditionally depicted as straight lines, which sometimes confuses people. Again, the point simply is that one slopes downward and the other upward.

Your key confusion is about what is on the horizontal axis. It's model complexity - not sample size. Yes, as you write, if we use some unbiased estimator, then increasing the sample size will reduce its variance, and we will get a better model. However, the bias-variance tradeoff is in the context of a fixed sample size, and what we vary is the model complexity, e.g., by adding predictors.

If model A is too small and does not contain predictors whose true parameter value is nonzero, and model B encompasses model A but contains all predictors whose parameter values are nonzero, then parameter estimates from model A will be biased and from model B unbiased - but the variance of parameter estimates in model A will be smaller than for the same parameters in model B.

$\endgroup$
3
  • 2
    $\begingroup$ Thank you for the answer. I mentioned $e^x$ solely to illustrate the point of obviously opposing functions. Either way, are you saying that the tradeoff is an attribute of multivariate systems and cannot be easily shown in the univariate case? Qualitatively speaking I get the point of complexity of the model vs overfitting but can it be shown mathematically? $\endgroup$
    – ayorgo
    Mar 24, 2018 at 11:49
  • 2
    $\begingroup$ You can show it mathematically if you restrict yourself to a specific model class, e.g., Ordinary Least Squares. In the simplest case, the true DGP may depend linearly on a single variable $x$. Model A would then be a simple mean model, and model B would be a regression on $x$, and you can calculate bias and variance. And if you want to, you can include higher powers of $x$ for even more variance. $\endgroup$ Mar 24, 2018 at 12:14
  • 1
    $\begingroup$ The values the OP mentions are population values. The estimates of these values may have non-zero correlation, e.g. King and Zhen: gking.harvard.edu/files/gking/files/0s.pdf see page 11 where they state "and so we are in the happy situation where reducing bias also reduces variance". However, as Stephan mentions the horizontal axis of the plot in the OP is model complexity and the example given by King and Zheng, is by default more complex than a logistic regression. $\endgroup$ Mar 24, 2018 at 14:06
4
$\begingroup$

Problems occur when a model $f(x,\theta)$ has a high tendency to fit the noise.

In that case the model tends to over-fit. That is, it is not only expressing the true model but also the random noise that you do not want to capture with your model (because the noise is a non-systematic part that does not allow you to make predictions for new data).

One might improve (reduce) the total error of fitting, by introducing some bias, when the this bias makes the variance/over-fitting reduce more strongly than the increase of the bias/under-fitting (ie not correctly representing the true model).

1. Why exactly $E[(\hat{\theta}_n - E[\hat{\theta}_n])^2]$ and $E[\hat{\theta}_n - \theta]$ cannot be decreased simultaneously?

This is not true. They can be decreased simultaneously (depending on the case). Imagine that you introduced some bias which both increased the variance as well as the bias. Then in the reverse direction reducing this bias will simultaneously reduce bias and variance.

For example a scaled root mean squared difference $c \sqrt{\frac{1}{n} {\sum(x_i-\bar{x})^2}}$ for sample of size $n$ is an unbiased estimator for the population standard deviation $\sigma$ when $c=\sqrt{\frac{n}{n-1}}$. Now, if you would have $c>\sqrt{\frac{n}{n-1}}$, then you would both reduce the bias as well as the variance when you reduce the size of this constant $c$.

However, the bias that is (intentionally) added in regularization is often of the kind that reduces the variance (e.g. you could reduce $c$ to a level below $\sqrt{\frac{n}{n-1}}$). Thus, in practice, you get a trade-off in bias versus variance and reducing the bias will increase the variance (and vice versa).

2. Why can't we just take some unbiased estimator and reduce the variance by increasing sample size?

In principle you can.

But,

  • This may require much more sampling effort which is expensive, and this is often a limitation.
  • Possibly there might also be computational difficulties with certain estimation problems and the sample size would need to increase extremely in order to solve this, if it is possible at all.

(e.g. high dimensionality parameters>measurements, or as in ridge regression: very shallow paths around the global optimum)

Often there is also no objection to bias. When it is about reducing the total error (as in many cases) then the use of a biased but less erroneous estimator is to be preferred.

About your counter example.

Related to your second question you can indeed reduce the error by increasing the sample size. And related to your first question you can also reduce both bias and variance (say you use a scaled sample mean $c\frac{\sum{x_i}}{n}$ as estimator of the population mean and consider varying the scaling parameter $c$).

However the region of practical interest is where the decreasing bias coincides with an increasing variance. The image below shows this contrast by using a sample (size = 5) taken from a normal distribution with variance = 1 and mean = 1. The unscaled sample mean is the unbiased predictor of the population mean. If you would increase the scaling of this predictor than you have both increasing bias and increasing variance. However if you decrease the scaling of the predictor then you have increasing bias, but decreasing variance. The "optimal" predictor is then actually not the sample mean but some shrinked estimator (see also Why is the James-Stein estimator called a "shrinkage" estimator?).

overfitting and underfitting in shrinking of sample mean

$\endgroup$
1
  • 1
    $\begingroup$ This explanation hits the nail exactly on the head and answers OP's confusion I believe. It is that some functions can reduce both at the same time, but due to regularization, most of the time we don't. Similarly with models that can use too many parameters such as NNs and overfit, this regularization is implicit in the architecture chosen. However there could still be a perfectly learnable function and in that case there may not be a bias-variance tradeoff. $\endgroup$ Aug 9, 2021 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.