2
$\begingroup$

Consider the regression model $$Y=X\beta+\epsilon$$ where the error terms in $\epsilon=(\epsilon_1,\ldots,\epsilon_n)^T$ are homoskedastic and where $n$ is the number of observations. Assume that we have an intercept and $k$ regressors; i.e., $\beta=(\beta_0,\ldots,\beta_k)$. Furthermore, denote the least squares estimate of $\beta$ by $\hat{\beta}$ and let $\hat{\epsilon}=Y-X\hat{\beta}$ be the residual.

Now, under the Guass-Markov assumptions $$\hat{\sigma}^2=\hat{\epsilon}^T\hat{\epsilon}/(n-k-1)\tag{1}$$ is an unbiased estimator of the variance $\sigma^2=E(\epsilon^2_i)$ of any error term $i$.

According to me the following holds: When I add regressors to my model, $k$ increases and consequently $1/(n-k-1)$ increases, while $\hat{\epsilon}^T\hat{\epsilon}$ decreases or stays constant (since when adding regressors we minimize the residual sum of squares over a larger domain). Hence, the effect of adding regressors on the estimated variance of the error term (i.e., $\hat{\sigma}^2$) is ambiguous.

However, I keep on hearing (e.g., by my professor in econometrics, in this lecture on multiple regression at page 23 and in two answers to a question here at CrossValidated) that the estimated variance of the error term decreases or stays constant when I increase the number of regressors. I do not agree $-$ am I missing something?

$\endgroup$
  • $\begingroup$ Look at the first comment in question you linked. You are confusing noise $\sigma^2$ which is part of the model specification and residuals $\hat{\epsilon} = Y - \hat{Y}$ $\endgroup$ – Łukasz Grad Mar 6 '17 at 21:14
  • $\begingroup$ Also $E(\hat{\epsilon}^T\hat{\epsilon}) = \sigma^2(n-k-1)$ not $\sigma^2I$ $\endgroup$ – Łukasz Grad Mar 6 '17 at 21:27
  • $\begingroup$ @ŁukaszGrad Sorry for the typo. In what way am I confusing the two? I am talking about the variance of the residuals (changed the title). The variance of the error terms are fixed, as they are assumed to be so. Note that some authors use the word 'error term' for what others calls 'residual'. $\endgroup$ – AnonymousIGuess Mar 6 '17 at 21:30
  • 2
    $\begingroup$ +1 This is a good question. It is closely related to the distinction between $R^2$ and the adjusted $R^2$ statistic. Although the former cannot decrease as variables are added to the model, the latter can decrease, precisely because the loss of a degree of freedom is not compensated by a concomitant decrease in the variance of $\hat\epsilon$. $\endgroup$ – whuber Mar 6 '17 at 21:48
  • 1
    $\begingroup$ @whuber Thanks! Yes, and $\hat{\sigma}^2=SST(1-R^2)/(n-k-1)$ as I understand it. $1-R^2$ stays constant or decreases when I add new regressors, while $SST$ (total sum of squares) is constant and $1/(n-k-1)$ increases. Hence the effect is ambigous. $\endgroup$ – AnonymousIGuess Mar 6 '17 at 22:03
1
$\begingroup$

Here's a little R simulation that confirms your intuition:

set.seed(9955)

N <- 10^3
p <- 100
X <- matrix(rnorm(N*p, mean=0, sd=1), N, p)
colnames(X) <- sprintf("x_%s", seq_len(ncol(X)))
y <- rnorm(N, mean=0, sd=1)  # True betas are zero
df <- as.data.frame(cbind(y, X))
names(df) <- gsub("V", "x_", names(df))

estimated_sigma_squared <- sapply(seq_len(p), function(k) {
    message("estimating model with constant and ", k, " Xs")
    model_formula <- reformulate(response="y", termlabels=sprintf("x_%s", seq_len(k)))
    model <- lm(formula=model_formula, data=df)
    sigma_squared_hat <- sum(residuals(model)^2) / (N - k - 1)
    return(sigma_squared_hat)
})

plot(estimated_sigma_squared, type="l")
any(diff(estimated_sigma_squared) > 0)  # True

A ggplot2 version of the plot:

sigma squared hat as function of number of regressors

Note that the maximum likelihood estimate of $\sigma^2$ has $N$ in the denominator (as opposed to $N-k-1$), and therefore cannot increase with $k$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.