3
$\begingroup$

As I understand it, in the specific context of linear regression, the R output "residual standard error" is an estimate of $\sigma$, the standard deviation of the distribution of the residuals. You can compute $\sigma^2$ by the MSE, which in general is the mean of the squared errors, but in regression the denominator is the residual degrees of freedom $n-p$ ($p$ representing the number of parameters including the intercept).

$$MSE=\frac{\sum_{i=1}^{n} y_i - \hat{y}_i}{n-p}$$

Why isn't residual standard error just called RMSE, when $\sigma^2$ is the MSE? Or is $\hat{\sigma}^2$ often called the residual variance (making the terms for $\sigma^2$ and $\sigma$ line up)?

$\endgroup$
5
  • $\begingroup$ I think this is widely regarded as a poor decision by the R developers. I’ve definitely seen discussion about it on here but can’t think of where. $\endgroup$
    – Dave
    Commented Mar 9, 2023 at 19:38
  • $\begingroup$ @Dave how would you call $\hat{\sigma}$ and $\hat{\sigma}^2$? $\endgroup$
    – fmtcs
    Commented Mar 9, 2023 at 19:47
  • $\begingroup$ Unbiased error variance estimate and the square root of the unbiased error variance estimate. (I might use “unbiased standard deviation” as slang, even though Jensen’s inequality shows such a term to be wrong (square root of an unbiased estimator is biased).) $\endgroup$
    – Dave
    Commented Mar 9, 2023 at 19:50
  • $\begingroup$ @Dave do you think calling $\hat{\sigma}^2$ the MSE is a mistake as well? $\endgroup$
    – fmtcs
    Commented Mar 9, 2023 at 19:53
  • 1
    $\begingroup$ Referring to “the” MSE is probably a mistake, since there are reasonable arguments for multiple calculations (an $n$ denominator and and $n-p$ denominator both make sense). I would want to define explicitly what I mean if there is any ambiguity about it. However, the R decision to call $\hat\sigma$ the standard error makes no sense to me, because standard errors are associated with parameters being estimated. To which parameter does $\hat\sigma$ correspond? (I don’t have an answer, which is why I have yet to see why R calls $\sigma$ a residual standard error.) $\endgroup$
    – Dave
    Commented Mar 9, 2023 at 19:57

1 Answer 1

2
$\begingroup$

Referring to “the” MSE is probably a mistake, since there are reasonable arguments for multiple calculations (an $n$ denominator and and $n−p$ denominator both make sense). I would want to define explicitly what I mean if there is any ambiguity about it.

However, the R decision to call $\hat\sigma$ the standard error makes no sense to me, because standard errors are associated with parameters being estimated. To which parameter does $\hat\sigma$ correspond? (I don’t have an answer, which is why I have yet to see why R calls $\hat\sigma$ a residual standard error.)

I do not really have a clean name for $\hat\sigma=\sqrt{ \frac{ \sum\left( y_i-\hat y_i \right)^2 }{ n-p} }$. While the expression inside the square root is unbiased for error variance (assuming fairly typical assumptions like the Gauss-Markov conditions), Jensen’s inequality means that $\hat\sigma$ is biased for the error standard deviation, so “unbiased error standard deviation” is not correct.

EDIT

The R developers appear to know that residual standard "error" is a misnomer and lament that it crept into the documentation for functions to such an extent that correcting this becomes difficult.

$\endgroup$
1
  • 1
    $\begingroup$ I recall seeing others on here lamenting this naming decision but cannot think offhand of where to find such posts. I welcome others to post links in the comments. $\endgroup$
    – Dave
    Commented Mar 18, 2023 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.