4
$\begingroup$

Consider the regression model $$Y=X\beta+\epsilon$$ where the error terms in $\epsilon=(\epsilon_1,\ldots,\epsilon_n)^T$ are homoskedastic and where $n$ is the number of observations. Assume that we have an intercept and $k$ regressors; i.e., $\beta=(\beta_0,\ldots,\beta_k)$. Furthermore, denote the least squares estimate of $\beta$ by $\hat{\beta}$ and let $\hat{\epsilon}=Y-X\hat{\beta}$ be the residual.

Now, under the Guass-Markov assumptions $$\hat{\sigma}^2=\hat{\epsilon}^T\hat{\epsilon}/(n-k-1)\tag{1}$$ is an unbiased estimator of the variance $\sigma^2=E(\epsilon^2_i)$ of any error term $i$.

According to me the following holds: When I add regressors to my model, $k$ increases and consequently $1/(n-k-1)$ increases, while $\hat{\epsilon}^T\hat{\epsilon}$ decreases or stays constant (since when adding regressors we minimize the residual sum of squares over a larger domain). Hence, the effect of adding regressors on the estimated variance of the error term (i.e., $\hat{\sigma}^2$) is ambiguous.

However, I keep on hearing (e.g., by my professor in econometrics, in this lecture on multiple regression at page 23 and in two answers to a question here at CrossValidated) that the estimated variance of the error term decreases or stays constant when I increase the number of regressors. I do not agree $-$ am I missing something?

$\endgroup$
6
  • $\begingroup$ Look at the first comment in question you linked. You are confusing noise $\sigma^2$ which is part of the model specification and residuals $\hat{\epsilon} = Y - \hat{Y}$ $\endgroup$ – Łukasz Grad Mar 6 '17 at 21:14
  • $\begingroup$ Also $E(\hat{\epsilon}^T\hat{\epsilon}) = \sigma^2(n-k-1)$ not $\sigma^2I$ $\endgroup$ – Łukasz Grad Mar 6 '17 at 21:27
  • $\begingroup$ @ŁukaszGrad Sorry for the typo. In what way am I confusing the two? I am talking about the variance of the residuals (changed the title). The variance of the error terms are fixed, as they are assumed to be so. Note that some authors use the word 'error term' for what others calls 'residual'. $\endgroup$ – Elias Mar 6 '17 at 21:30
  • 2
    $\begingroup$ +1 This is a good question. It is closely related to the distinction between $R^2$ and the adjusted $R^2$ statistic. Although the former cannot decrease as variables are added to the model, the latter can decrease, precisely because the loss of a degree of freedom is not compensated by a concomitant decrease in the variance of $\hat\epsilon$. $\endgroup$ – whuber Mar 6 '17 at 21:48
  • 1
    $\begingroup$ @whuber Thanks! Yes, and $\hat{\sigma}^2=SST(1-R^2)/(n-k-1)$ as I understand it. $1-R^2$ stays constant or decreases when I add new regressors, while $SST$ (total sum of squares) is constant and $1/(n-k-1)$ increases. Hence the effect is ambigous. $\endgroup$ – Elias Mar 6 '17 at 22:03
1
$\begingroup$

Here's a little R simulation that confirms your intuition:

set.seed(9955)

N <- 10^3
p <- 100
X <- matrix(rnorm(N*p, mean=0, sd=1), N, p)
colnames(X) <- sprintf("x_%s", seq_len(ncol(X)))
y <- rnorm(N, mean=0, sd=1)  # True betas are zero
df <- as.data.frame(cbind(y, X))
names(df) <- gsub("V", "x_", names(df))

estimated_sigma_squared <- sapply(seq_len(p), function(k) {
    message("estimating model with constant and ", k, " Xs")
    model_formula <- reformulate(response="y", termlabels=sprintf("x_%s", seq_len(k)))
    model <- lm(formula=model_formula, data=df)
    sigma_squared_hat <- sum(residuals(model)^2) / (N - k - 1)
    return(sigma_squared_hat)
})

plot(estimated_sigma_squared, type="l")
any(diff(estimated_sigma_squared) > 0)  # True

A ggplot2 version of the plot:

sigma squared hat as function of number of regressors

Note that the maximum likelihood estimate of $\sigma^2$ has $N$ in the denominator (as opposed to $N-k-1$), and therefore cannot increase with $k$.

$\endgroup$
2
$\begingroup$

starting from: $\hat{\sigma}^{2} = \frac{\hat{\epsilon}^{T}\hat{\epsilon}}{(n-k)}$ Where $\epsilon^T\epsilon$ denotes the residuals, and $\sigma^2$ denotes the residual variance (i.e the variance of that’s not explained by your regressors), if your additional regressor do not gives any additional informations ((has not explanatory power))this will lead to an increase in the variance due to the fact that the denominator decreases. On the other hand if the additional regressor gives some additional informations to the model (has explanatory power) the residuals will decreases (this will overcome the increase due to the denominator).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.