Suppose that we are interested in estimating a quantity $\theta$ with an estimator $\hat{\theta}$, and seek to minimize error in the $l_2$ sense, so our loss is $||\theta - \hat{\theta}||_2$.

It is well known that this loss may be decomposed into the sum of squared bias and variance of the estimator. It is also well known that incurring bias can lead to higher losses in variance and thus lower Mean Squared Error, or MSE, overall.

However, the examples of this that I have been exposed to have all decreased the variance of an estimator by making it smaller on average in the absolute value sense.

For instance, if $X_i \sim N(\mu, \sigma^2)$, $i = 1:N$ and we're interested in inference on $\sigma^2$, we could consider two estimators:

$\hat{\sigma}^2_1 = \frac{\sum (x_i - \bar{x})^2}{n-1}$

$\hat{\sigma}^2_2 = \frac{\sum (x_i - \bar{x})^2}{n}$

$\hat{\sigma}^2_1$ is the canonical estimate for variance, sometimes referred to as 'sample standard deviation' in introductory contexts. It is unbiased. $\hat{\sigma}^2_2$, will always be a little bit smaller than $\hat{\sigma}^2_1$, and so will have bias. However, the concomitant decrease in variance outweighs the increase in squared bias, and it can be shown that:

$MSE(\hat{\sigma}^2_1) \ge MSE(\hat{\sigma}^2_2) \forall n, \sigma^2$

Which estimator is best is a discussion I'd like to set aside for now. In this example, and others that I've seen, bias is incurred because we are shrinking a parameter towards zero. To my understanding, this is also how things like $l_p$ regularization, such as lasso and ridge regression, work in supervised problems.

Does an example exist where bias is introduced to an estimator to decrease MSE (or, any suitable loss function) that could not be described as shrinkage? Or is shrinkage the only way this can be accomplished?

  • $\begingroup$ I assume that in your second paragraph you mean "... incurring bias can lead to LOWER losses" instead of "... higher losses"? $\endgroup$ – jbowman Jul 18 '17 at 19:53
  • $\begingroup$ Are you restricting yourself to situations where you have an unbiased estimator in hand and seek to reduce the MSE by some post-processing step, or are you also considering a comparison of two estimators, one of which is biased and the other of which is not? $\endgroup$ – jbowman Jul 18 '17 at 19:54
  • $\begingroup$ Hey @jbowman, to answer comment 1, perhaps my phrasing is awkward, I mean higher losses in variance. To answer comment 2, no, I am not restricting to situations where one estimator is a perturbation of another, though I admit I convey this poorly. $\endgroup$ – John Madden Jul 18 '17 at 20:57
  • $\begingroup$ ... I'm not seeing how incurring bias where there was none before + higher (losses in) variance can lead to lower MSE overall. $\endgroup$ – jbowman Jul 18 '17 at 21:01
  • $\begingroup$ @jbowman I think you're getting caught up in the phrasing. I mean that increasing bias can sometimes decrease variance enough to lower MSE, nothing more, nothing less, by that phrase. $\endgroup$ – John Madden Jul 18 '17 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.