5
$\begingroup$

I have acquired the impression that many of the people doing statistical work, will prefer a biased estimator $\hat b$ to an unbiased one $\hat \beta$, if the former has lower Mean Squared Error.
This appears to be a way to quantify the "bias-variance trade off" (excerpt from the CV tag of the same name):

...In predictive modeling, unbiased models can have higher variance, & thus be less accurate. Modelers may prefer some bias to maximize accuracy...

...since, in order to have lower MSE, it means that the variance is not just lower, but lower below a specific threshold,

$$MSE(\hat b) < MSE(\hat \beta) \implies \text{Var}(\hat b) < \text{Var}(\hat \beta) - \text{Bias}^2(\hat b)$$

My request is for an elaboration of the argument as to why MSE is a reasonable quantifier of the "bias-variance tradeoff", and related references hopefully.

...or against, proposing some other measure.

I link here two threads, which provide arguments in favor of using MSE or more generally "squared differences" as "loss functions",

Why is the squared difference so commonly used?

and one that I have also contributed,

What makes mean square error so good?

but I stress that I am mostly interested in whether there is some argument about why MSE is a good quantifier of the bias-variance trade off.

$\endgroup$

1 Answer 1

1
$\begingroup$

I think you're thinking of this in the wrong way. It isn't that MSE quantifies bias-variance tradeoff. It's that people care about MSE, and MSE can be expressed as in terms of bias and variance. Because of that, people are willing to accept a bit of bias in exchange for a large reduction in variance, since that reduces the quantity they care about (MSE).

$\endgroup$
3
  • $\begingroup$ That's a bit circular don't you think? Why people care about MSE? As for the"large" reduction, it depends: if the bias is lower than unity, an estimator that had variance lower than the variance of the unbiased estimator minus the bias - not bias squared, would be even more preferable. So the question is: "If I am better in MSE terms, it means I have variance lower by at least the squared bias". Fine, ok. Why reducing the variance by an amount equal to squared bias is the criterion to go by? $\endgroup$ Jul 25, 2019 at 22:03
  • $\begingroup$ It isn't circular. People care about MSE. Since MSE has the bias-variance decomposition, people can manipulate bias and variance to lower the MSE. $\endgroup$
    – Dave
    Jul 25, 2019 at 22:10
  • 1
    $\begingroup$ Again: why people care about MSE specifically? $\endgroup$ Jul 25, 2019 at 22:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.