In linear modelling, why do we tend to use the unbiased estimator $\hat{\sigma}^2=\frac{RSS}{n-p}$ instead of the MLE estimator $ \hat{\sigma}^2 = \frac{RSS}{n}$?

I understand it's most likely because it is unbiased, and the MLE estimator is not, but why does this benefit us? Are there general rules for deciding which estimator to use?

  • 2
    $\begingroup$ This Q&A related and possibly relevant. Unbiasedness is probably a big part of the reason. Also, modifications in traditional uses of chi-squared and F-distributions would be necessary before using MLEs because formulas are keyed to the unbiased estimators. $\endgroup$
    – BruceET
    Commented Oct 24, 2020 at 21:25
  • $\begingroup$ I’ll also add that a primary goal of linear regression is inference, not prediction. Those hypothesis tests matter a lot. $\endgroup$
    – Dave
    Commented Oct 24, 2020 at 21:59
  • 2
    $\begingroup$ For the purpose of the usual hypothesis tests, there's no difference whatsoever between the two formulations. (If you were to use the MLE, you would have to adjust the t and F statistics to compensate. On the other hand, if you are using the MLE for $\hat\sigma^2$ then you're probably not even doing t or F tests, but instead are relying on the asymptotic tests for sufficiently large $n,$ indicating there's little difference between $n$ and $n-p$ anyway.) $\endgroup$
    – whuber
    Commented Oct 24, 2020 at 22:05
  • $\begingroup$ One subtle issue here is that at least for estimation in a simple normal setup without regression, the ML $\hat\sigma^2$ with factor $\frac{1}{n}$ has a smaller MSE than the standard one with factor $\frac{1}{n-1}$. I'd think this also holds for regression although I haven't seen a proof (it should be fairly easy to show). Now this seems to favour the ML estimator, however I'd argue that the MSE is somewhat inappropriate for variance estimation, because asymmetry makes underestimation, particularly when getting close to 0, worse than overestimation by the same amount. $\endgroup$ Commented Oct 25, 2020 at 10:08
  • 1
    $\begingroup$ I wonder whether anyone has set up an appropriate (how to define that?) asymmetric objective function as alternative to the MSE - the standard estimator with factor $\frac{1}{n-p}$ may look better in this respect, but I haven't seen such a thing. $\endgroup$ Commented Oct 25, 2020 at 10:10


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.