Does higher degrees of freedom reduce variance in linear regression?

Question

I'm trying to figure out why higher degrees of freedom $(n-1-k)$ in a linear regression is "better". I can't see how higher df would automatically result in lower $MSE$, since every new df (data point) increases the sum of squared errors in the numerator, just as it increases $n-1-k$ in the denominator. It would make sense if there was some variance measure like $MSE / n$, where higher df would unambiguously reduce variance, just like for a univariate distribution where increasing df reduces the variance of the sample mean.

In linear regression, if you force the slope coefficient to be $0$, it reduces to a univariate model, $Y = \beta_0 + error$. $\beta_0$ is the sample mean of the $y$ values. The sample variance = sum of squared errors / $(n-1)$. And the variance of the sample mean = sample variance / $n$.

Now say you allow the slope parameter to be nonzero, i.e. $Y = \beta_0 + \beta_1*X + error$. The mean squared error of the estimate ($MSE$) = sum of squared errors / $(n-1-k)$. From this, I assume $MSE$ is analogous to sample variance in the above no‐slope model, since they both refer to the variance of the error term.

So is there some concept for linear regression like the variance of the sample mean error = $MSE$ / $n$? Something that would be analogous to the variance of the sample mean = sample variance / $n$ (in a univariate setting)? Or is it meaningless because the mean residuals of the regression in any sample will by definition be equal to $0$, i.e. the expected value of the error?

Answer 1

The formula for the variance of a regression slope coefficient is:

$$ Var(\hat{\beta}) = \frac{\sigma^2 }{\sum (x_{i}-\bar{x})^2} $$

where $\sigma^2$ is estimated by the MSE:

$$ \hat{\sigma}^2 = \frac{1}{n-k-1}\sum \hat{u}^2_i $$

In textbooks, it is typically written that a larger sample size results in a smaller variance of regression coefficients, but that statement only considers the denominator of the variance formula. As the sample size increases, so does the total variation in the explanatory variable(s), which in turn reduces the variance.

A larger sample size is not necessarily going to reduce MSE (or the variances of betas, which rely on the MSE formula for their calculation). Variances of betas are directly proportional to MSE (as seen above, MSE is the numerator in the formula for deriving the variance of regression slope), so the larger the MSE (error variance), the larger the variance of regression slope will be. Assuming you have a random sample, additional data points can make MSE smaller or larger, so it is possible for it to fluctuate (though it will converge to some population value eventually).

Answer 2

I figured this out thanks to the help I received on a related question. In case anyone is curious. The short answer is that higher df does not reduce MSE directly. But higher df reduces the variance of thr estimated coefficient beta0_hat in the regression, which is MSE/n, thus reducing variance of the estimated model Y_hat, which is the sum of beta0_hat and beta1_hat variance. Because the variance of Y is just the variance of Y_hat + variance of the error, i.e. MSE, higher df also narrows the variance of Y (corresponding to prediction interval width), but this flows through the lower variance of the beta0_hat = MSE/n component, not directly through the other MSE component arising from the error variance.

In the univariate analogy, the slope coefficient is forced to be 0, so beta0_hat reduces to the sample mean, and variance of the sample mean is s^2/n, while variance of the error term is s^2. In a prediction interval, the variance used is s^2/n + s^2. The first term arises from uncertainty around the sample mean (this goes to 0 if you know the true population mean). The second term arises from the error (difference between observations and the model, and in this case the model is just the sample mean). This second term goes to 0 if you are estimating the variance of the model i.e. the variance of the sample mean.