I am checking other's regression analysis work on a $p > n$ data. I can only see the results but not the process how he did it.

I believe he made mistakes. Since $p > n$, $X^T X$ is not full rank it is not invertible. So we cannot find the OLS coefficient.

However, I am not sure my reasoning is correct. I have not used linear algebra for a long time.


  1. No penalized methods involved.
  2. He used stepwise regression for variable selection. A new question: would such algorithm stop if number of variables in the model equal to the number of sample points?
  3. His goal is to find out which variables are important. Doesn't care about the prediction power.
  • 5
    $\begingroup$ Yes you are correct in that $X^TX$ is non-invertible. However, perhaps your colleague performed some penalized regression like LASSO to get coefficients? $\endgroup$
    – ilanman
    Oct 20 '16 at 17:44
  • 1
    $\begingroup$ What are $p$ and $n$? $\endgroup$ Oct 20 '16 at 18:21
  • $\begingroup$ A bit more context may be useful. Is the goal forecasting? Is it consistently estimating some effect $b_i$? Do you know if your colleague performed LASSO or ridge regression? $\endgroup$ Oct 20 '16 at 18:30
  • $\begingroup$ @RodrigodeAzevedo $p$ and $n$ are classically the number of covariates and of independent observations. IE, $X$ is $n \times p$. $\endgroup$
    – Elvis
    Oct 20 '16 at 19:07
  • 1
    $\begingroup$ If he's using forward stepwise regression, he'd never have more than $n$ variables in the actual regression matrix used (starting from 1 = the intercept, finding the best variable to add incrementally.) That's not to say forward stepwise regression (or any stepwise regression) is good, it's just to say it would have avoided the identifiability problem. $\endgroup$
    – jbowman
    Oct 20 '16 at 19:44

If $\mathrm X$ is $n \times p$ and $p > n$, then it is fat and, thus,

$$\mbox{rank} (\mathrm X) = \mbox{rank} (\mathrm X^T \mathrm X) \leq n < p$$

Hence, $\mathrm X^T \mathrm X$ does not have full rank and, thus, it is not invertible.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.