1
$\begingroup$

If $ \hat{\beta} = (X'X)^{-1}X'y $ with $ X $ being an $ n \times k $ matrix, then as I understand it, as long as $ k \leq n $, $ X'X $ is invertible (as long as all other OLS assumptions are fulfilled), and thus there is a unique solution for $ \hat{\beta} $. But if there is a perfect fit with $ k = n $, is it then true that $ \hat{\sigma}(u) = 0 $ as $ \hat{\sigma}(u) = \hat{s}^2 = \frac{1}{n-k+1} \epsilon'^\top \epsilon = 0 $ and thus $ \text{Var}(\hat{\beta}) = \hat{\sigma}(u) (X'X)^{-1} $ is also $ 0 $?

$\endgroup$
1
  • 2
    $\begingroup$ McElreath has a nice illustration of how fitting an $n-1$ order polynomial results in a perfect fit with $R^2 = 1$ and thus no unexplained (residual) variance. $\endgroup$
    – Durden
    Commented Mar 15 at 19:20

1 Answer 1

5
$\begingroup$

One premise of your question (the thing you describe as your understanding) is incorrect.

Even with $k<<n$ it's perfectly possible for the columns of X (plus constant) to have some linear combination that's zero, whence $X'X$ will not be invertible.

is it then true that $σ(u)=0$

No, the population variance of the error will not be zero.

If you are asking about some estimate of it (which we might call $s$, say), what's the form of your estimator? (Pay particular attention to the denominator)

$\endgroup$
5
  • $\begingroup$ Yes, that's correct. The question lacks precision. It's entirely possible, as you mentioned. While the population variance isn't zero, it's not directly observable. Therefore, when calculating the variance of the estimates of ß, we rely on the sample's error term variance. With a perfect fit in the regression, the variance of the estimated residuals becomes zero, implying that sigma(u) is also zero as it is estimated based on the smaple. Hence, the variance of ß is also zero, but is it really? $\endgroup$
    – Joe94
    Commented Mar 15 at 0:51
  • $\begingroup$ AGAIN: What is the denominator on your estimate of the variance. Please don't avoid the thing I was very careful to draw your attention to. $\endgroup$
    – Glen_b
    Commented Mar 15 at 2:03
  • $\begingroup$ Thanks for the response again. So $\hat{s}^2 = \frac{1}{n-k+1} \epsilon'^\top \epsilon$ ... but $\epsilon'^\top \epsilon$ is still zero, isnt it? $\endgroup$
    – Joe94
    Commented Mar 15 at 18:37
  • $\begingroup$ Unlikely, because that's a sum of squares. But it comes down to what you mean by "$\epsilon$" and "$u,$" because it appears you might not be carefully distinguishing between errors and residuals. $\endgroup$
    – whuber
    Commented Mar 15 at 20:36
  • $\begingroup$ @Joe94 Be very careful what $k$ means in your question where you state $k=n$, and here, where you say your denominator is $n-k+1$. You don't seem to be consistent across the two (if $X$ is $n\times k$, what's the "+1" for?). If you are more careful about how you do it, you'll see that in fact you (almost certainly) mean to divide by $n-k$ and hence that $s^2 = 0/0$. $\endgroup$
    – Glen_b
    Commented Mar 16 at 3:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.