Suppose we have that $n << p$ and that we have a linear regression model where the predictors outnumber the observations. In this case, usually in statistical software packages, the standard errors and hence the confidence intervals show as NaN. The formal standard error formula is given by:

$$ s_{\hat\beta_j}=\sqrt{\frac{\sum_i\hat\epsilon_i^2}{(n-p-1)\sum_i(x_i-\bar x)^2}} $$

It seems here that the only thing preventing the estimate is the $(n-p-1)$ on the bottom causing an invalid negative value. I understand it is there due to unbiasedness reasons. However, why can't be just remove $(n-p-1)$ from the term otherwise?

  • 1
    $\begingroup$ You cannot generally create a valid formula just by removing the part that offends you. The ludicrousness of dividing by $n-p-1$ when $p \ge n-1$ is a strong signal that something is fundamentally wrong in that case. Indeed, assuming the predictors (plus the constant vector) are of full rank, you can perfectly fit the response, leaving no way to estimate the error. Reporting NaN sounds like the perfect way to express that ineluctable fact. $\endgroup$ – whuber Oct 17 '17 at 13:23
  • $\begingroup$ As whuber says you get a perfect fit which means the standard error is 0. $\endgroup$ – Michael Chernick Oct 17 '17 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.