Let $p$ be the number of parameters in a linear regression model, let $n$ be the number of observations, and let $p>n$.

$$\mathbb E[Y\vert X] = \beta_0 +\beta_1X_1 +...+\beta_pX_p$$

Does the Gauss-Markov theorem still say that the OLS solutions (under the usual conditions for Gauss-Markov to apply) for the $\beta$ vector are the best linear unbiased estimators? I assume that the OLS solutions give equal variance and all are unbiased, but can some other linear and unbiased estimator achieve lower variance?

  • $\begingroup$ The $p\gt n$ case is equivalent to the $p\le n$ case because of the rank deficiency of the system. However, two or more of the $\beta_i$ cannot be estimated at all, for the same reason. $\endgroup$
    – whuber
    May 12 at 16:14
  • $\begingroup$ @whuber That seems like a contradiction. If the $p>n$ case is equivalent to $p<n$, then how can $p>n$ mean that some $\beta_i$ cannot be estimated when we estimate them just fine when $p<n?$ $\endgroup$
    – Dave
    May 12 at 16:26
  • 1
    $\begingroup$ The equivalence concerns the model's fit. As you know, the meaning of any coefficient in a multiple regression depends on all the other explanatory variables that are included, so no direct comparison is possible between the original (redundantly specified) model and the model with superfluous variables removed. Basically, your $p\gt n$ model uses too many parameters and you just cannot identify them all from the data alone. $\endgroup$
    – whuber
    May 12 at 17:05


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.