38
$\begingroup$

I'm surprised this hasn't been asked before, but I cannot find the question on stats.stackexchange.

This is the formula to calculate the variance of a normally distributed sample:

$$\frac{\sum(X - \bar{X}) ^2}{n-1}$$

This is the formula to calculate the mean squared error of observations in a simple linear regression:

$$\frac{\sum(y_i - \hat{y}_i) ^2}{n-2}$$

What's the difference between these two formulas? The only difference I can see is that MSE uses $n-2$. So if that's the only difference, why not refer to them as both the variance, but with different degrees of freedom?

Improve this question
$\endgroup$
7
  • $\begingroup$ What is it about the wikipedia page here that is not clear? $\endgroup$
    – TrynnaDoStat
    Mar 5, 2015 at 19:32
  • 5
    $\begingroup$ Variance is the average of squared deviation of the observations from the mean. The MSE in contrast is the average of squared deviations of the predictions from the true values. $\endgroup$
    – random_guy
    Mar 5, 2015 at 19:38
  • 3
    $\begingroup$ Both "variance" and "mean squared error" have multiple formulas and varying applications. To clarify your question, could you (a) describe what kind of data you are applying these concepts to and (b) give formulas for them? (It's likely that in so doing you will discover the answer to your question, too.) $\endgroup$
    – whuber ♦
    Mar 5, 2015 at 19:41
  • 8
    $\begingroup$ There's a more general formula, which both are special cases of: $\frac{\sum_i(y_i-\hat{y}_i)^2}{n-p}$ where $p$ is the number of parameters estimated in obtaining $\hat{y}$ $\endgroup$
    – Glen_b
    Mar 6, 2015 at 3:05
  • $\begingroup$ @Glen_b could you please provide a reference for more information on this general formula? $\endgroup$
    – trianta2
    Nov 12, 2019 at 4:08

2 Answers 2

Reset to default
38
$\begingroup$

The mean squared error as you have written it for OLS is hiding something:

$$\frac{\sum_{i}^{n}(y_i - \hat{y}_i) ^2}{n-2} = \frac{\sum_{i}^{n}\left[y_i - \left(\hat{\beta}_{0} + \hat{\beta}_{x}x_{i}\right)\right] ^2}{n-2}$$

Notice that the numerator sums over a function of both $y$ and $x$, so you lose a degree of freedom for each variable (or for each estimated parameter explaining one variable as a function of the other if your prefer), hence $n-2$. In the formula for the sample variance, the numerator is a function of a single variable, so you lose just one degree of freedom in the denominator.

However, you are on track in noticing that these are conceptually similar quantities. The sample variance of $y$ measures the spread of the data around the sample mean of $y$ (in squared units), while the MSE measures the vertical spread of the data around the sample regression line (in squared vertical units).

Improve this answer
$\endgroup$
5
  • $\begingroup$ @amoeba Hey! Thanks for the attention. Is there an official CV style guide that prompted this edit? If so I wanna learn of it. If not, well, Glen_b once rightly admonished me for being colonizing with my personal style preferences and edits to others Qs and As. What do you think? (And I ask this in a collegial tone: I think your edit does add something. Just wanna understand our editing values better.) $\endgroup$
    – Alexis
    Mar 7, 2015 at 15:10
  • 1
    $\begingroup$ I don't think there is any official CV style guide making this suggestion, but in LaTeX there are inline formulas (marked with one dollar sign) that are rendered directly in the block of text, and displayed formulas (marked with two dollar signs) that are rendered on a separate line. Displayed formulas use different layout. Your formula was originally on a separate line but marked with one dollar sign; I don't think this makes sense. However, you are right about personal preferences, so feel free to roll back with apologies. The reason I edited was that I was fixing a typo in the Q anyway. $\endgroup$
    – amoeba
    Mar 7, 2015 at 15:23
  • 1
    $\begingroup$ if there is no intercept term $\beta_0$ in the regression problem, then the degrees of freedom of MSE is equal to $n-1$ like in the variance formula instead of $n-2$ $\endgroup$
    – develarist
    Nov 29, 2019 at 15:46
  • $\begingroup$ What is the reason why we use MSE as an estimator for the variance, instead of the sample variance? Arn't both of them unbiased estimators of the variance? $\endgroup$
    – woowz
    Aug 12, 2022 at 21:51
  • $\begingroup$ @woowz The variance of what? The 'sample variance' is typically interpreted as being of a single variable, the MSE is a sample variance of $y$ about the regression line, so those are conceptually distinct. $\endgroup$
    – Alexis
    Aug 14, 2022 at 5:01
1
$\begingroup$

In the variance formula, the sample mean approximates the population mean. The sample mean is calculated for a given sample with $n$ data points. Knowing the sample mean leaves us with only $n-1$ independent data points as the $n$th data point is constrained by the sample mean, so ($n-1$) degrees of freedom (DOF) in the denominator in the variance formula.

To get the estimated value of y ($= \beta_{0} + \beta_{1}\times x$) in the MSE formula, we need to estimate both $\beta_{0}$ (i.e. the intercept) as well as $\beta_{1}$ (i.e. the slope) so we lose 2 DOF, and so that is the reason for ($n-2$) in the denominator in the MSE formula.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.