6
$\begingroup$

For some estimator $\hat{\theta}$, we have the "bias-variance trade-off": $MSE(\hat{\theta}) = bias^2(\hat{\theta}) + var(\hat{\theta}).$

When I think of a trade-off, I would expect that as the variance goes down, the squared bias would go up, and vice versa. But I'm not sure this is the case here, since $MSE(\hat{\theta})$ is not fixed as $\hat{\theta}$ as varies.

Example: Consider two silly estimates for the mean of a single normally distributed data point $x \sim N(\theta, 1)$: $\delta_0(x) = 0$ and $\delta_1(x) = 1$. Both of these has the same variance of $0$. Furthermore, $bias^2(\delta_0) = \theta^2$ and $bias^2(\delta_1) = (1-\theta)^2$. Therefore, both estimators have the same bias but different squared biases (assuming $\theta \ne \frac{1}{2}$).

So in what sense is there a trade-off here?

Improve this question
$\endgroup$
6
  • 4
    $\begingroup$ The bias-variance tradeoff is a "rule of thumb" and not a law. No one claimed it was a law. $\endgroup$
    – Jake Westfall
    Jun 16, 2017 at 20:28
  • 1
    $\begingroup$ @JakeWestfall could you elaborate? $\endgroup$
    – Mark White
    Jun 16, 2017 at 20:33
  • 4
    $\begingroup$ @Jake Isn't the concept of a "minimum variance unbiased estimator" tantamount to such a law? Its very existence shows that if one can find any estimator (in the same family) with lower variance, it must increase the bias above zero, right? Qman: Your $\delta_i$ are not written like estimators. Estimators are functions of the data, not of the parameter. It isn't at all clear how you are computing the "bias" or even what you might mean by it. $\endgroup$
    – whuber ♦
    Jun 16, 2017 at 21:07
  • $\begingroup$ Right. I think the $\delta$'s should be functions of $x$, not $\theta$. $\endgroup$
    – theQman
    Jun 16, 2017 at 21:30
  • 4
    $\begingroup$ It's possible for an estimator to be dominated by another in both bias and variance simultaneously. This doesn't contradict the fact that there is a trade-off between those that are on the "boundary" of the feasible set in bias-variance space (the ones that achieve the lowest variance for a fixed level of bias, like MVUE). $\endgroup$
    – Chris Haug
    Jun 16, 2017 at 23:09

1 Answer 1

Reset to default
2
$\begingroup$

I share your skepticism that there is a tradeoff. A typical way to think about the bias-variance decomposition of MSE, such as in regularized regression, is that we accept a bit of bias in our estimator in exchange for a large reduction in variance. However, we do this to achieve lower MSE, not to maintain the MSE. Thus, while I understand the "trade" being used to describe trading your unbiased, high variance estimator for a slightly biased, low variance estimator, "tradeoff" to me implies keeping MSE constant, and I try not to describe it as a "tradeoff", preferring to refer to a bias-variance "decomposition".

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.