2
$\begingroup$

I'm trying to figure out why higher degrees of freedom $(n-1-k)$ in a linear regression is "better". I can't see how higher df would automatically result in lower $MSE$, since every new df (data point) increases the sum of squared errors in the numerator, just as it increases $n-1-k$ in the denominator. It would make sense if there was some variance measure like $MSE / n$, where higher df would unambiguously reduce variance, just like for a univariate distribution where increasing df reduces the variance of the sample mean.

In linear regression, if you force the slope coefficient to be $0$, it reduces to a univariate model, $Y = \beta_0 + error$. $\beta_0$ is the sample mean of the $y$ values. The sample variance = sum of squared errors / $(n-1)$. And the variance of the sample mean = sample variance / $n$.

Now say you allow the slope parameter to be nonzero, i.e. $Y = \beta_0 + \beta_1*X + error$. The mean squared error of the estimate ($MSE$) = sum of squared errors / $(n-1-k)$. From this, I assume $MSE$ is analogous to sample variance in the above no‐slope model, since they both refer to the variance of the error term.

So is there some concept for linear regression like the variance of the sample mean error = $MSE$ / $n$? Something that would be analogous to the variance of the sample mean = sample variance / $n$ (in a univariate setting)? Or is it meaningless because the mean residuals of the regression in any sample will by definition be equal to $0$, i.e. the expected value of the error?

$\endgroup$
1
$\begingroup$

The formula for the variance of a regression slope coefficient is:

$$ Var(\hat{\beta}) = \frac{\sigma^2 }{\sum (x_{i}-\bar{x})^2} $$

where $\sigma^2$ is estimated by the MSE:

$$ \hat{\sigma}^2 = \frac{1}{n-k-1}\sum \hat{u}^2_i $$

In textbooks, it is typically written that a larger sample size results in a smaller variance of regression coefficients, but that statement only considers the denominator of the variance formula. As the sample size increases, so does the total variation in the explanatory variable(s), which in turn reduces the variance.

A larger sample size is not necessarily going to reduce MSE (or the variances of betas, which rely on the MSE formula for their calculation). Variances of betas are directly proportional to MSE (as seen above, MSE is the numerator in the formula for deriving the variance of regression slope), so the larger the MSE (error variance), the larger the variance of regression slope will be. Assuming you have a random sample, additional data points can make MSE smaller or larger, so it is possible for it to fluctuate (though it will converge to some population value eventually).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks AlexK - as you noted, the variance of the slope coefficient does not vary with n, apart from the $\sigma^2$ or MSE term. I was thinking more of the confidence interval of the average predicted Y given some x0, or the prediction interval of a new data point given some x0. In both those formulas, in addition to the variance of the slope coefficient you mentioned, there is also a component relating to the variance of the intercept coefficient, estimated by $s^2/n$ or MSE/n. Thus, as df and n increase, the confidence and prediction intervals should tighten due to that 1/n term, right? $\endgroup$ – Guest Jun 23 at 6:49
  • 1
    $\begingroup$ The confidence interval will shrink with larger $n$, but the prediction interval may not. See this answer: stats.stackexchange.com/a/231645/241093. $\endgroup$ – AlexK Jun 23 at 18:05
  • 1
    $\begingroup$ And just to clarify, variance of the slope coefficient varies with $n$ also because of the denominator in the variance formula. As $n$ increases, there is more variation in the explanatory variable (unless all values are equal, which is unlikely), so the denominator will grow and variance will shrink. $\endgroup$ – AlexK Jun 23 at 18:29
  • $\begingroup$ So as n goes to infinity, the end term in a CI $\sqrt{1/n + \frac{(x-\bar x)^2}{\sum (x_i - \bar x)^2}}$ goes to 0, while the end term in a PI $ \sqrt{1+ 1/n + \frac{(x-\bar x)^2}{\sum (x_i - \bar x)^2}}$ goes to 1. That's why the width of the CI converges to $0 * MSE = 0$ while the width of the CI PI converges to $1 * MSE = MSE$? $\endgroup$ – Guest Jun 23 at 21:57
  • 1
    $\begingroup$ Yes, except $MSE$ is under the square root and there is also a t-stat in the width formula. Look at the last formula in the linked answer. $\endgroup$ – AlexK Jun 23 at 22:41
0
$\begingroup$

I figured this out thanks to the help I received on a related question. In case anyone is curious. The short answer is that higher df does not reduce MSE directly. But higher df reduces the variance of thr estimated coefficient beta0_hat in the regression, which is MSE/n, thus reducing variance of the estimated model Y_hat, which is the sum of beta0_hat and beta1_hat variance. Because the variance of Y is just the variance of Y_hat + variance of the error, i.e. MSE, higher df also narrows the variance of Y (corresponding to prediction interval width), but this flows through the lower variance of the beta0_hat = MSE/n component, not directly through the other MSE component arising from the error variance.

In the univariate analogy, the slope coefficient is forced to be 0, so beta0_hat reduces to the sample mean, and variance of the sample mean is s^2/n, while variance of the error term is s^2. In a prediction interval, the variance used is s^2/n + s^2. The first term arises from uncertainty around the sample mean (this goes to 0 if you know the true population mean). The second term arises from the error (difference between observations and the model, and in this case the model is just the sample mean). This second term goes to 0 if you are estimating the variance of the model i.e. the variance of the sample mean.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.