I'm running a PCA over a data set of $N \times p$ size ($N\approx 1000$ being the number of measurements and $p\approx 200$ being the number of dimensions/predictors).

I expect many of the predictors to be correlated and that the dimensions can consequently be reduced. I can even drop some columns that are linearly dependent with respect to the others.

When I run the PCA I find that $\sim 50\%$ of the variance can be explained by the first 5 PCs, suggesting that the predictors can actually be grouped.

But I am concerned about the smallness of the correlation matrix ($R$) determinant, which is $\det(R) \approx 10^{-100}$ or a ridiculous number like that.

Do the results make sense with such a small number?

Moreover, I see that the PCA results change (a lot!) if I round the input numbers to drop non-relevant digits, like the 10th digit or so. I think this is linked with the fact we are working with such a small determinant.

Since a small determinant in R indicates that there are redundant dimensions, I would say that the PCA is the way to go to reduce them. Nevertheless, does it make sense to run a PCA with such a small determinant? If not, what is the best way to reduce the dimensionality of the problem?

  • $\begingroup$ Singularity is not an obstacle to PCA. As for whether to remove the highly correlated variables (note btw that singularity is not a synonym to the full correlatedness) or not - you should think over what you want in the end, for it will affect the results. $\endgroup$ – ttnphns Mar 9 '16 at 8:20
  • $\begingroup$ I am ok with removing the highly correlated variables. It's just that I would expect it to be done by the PCA itself. On the other hand I don't know if it is legal to run a PCA over a singular correlation matrix. Are you saying that it is ok? [Here] (google.it/…) I read that I should check that the determinant is greater than 10^-5 ... $\endgroup$ – Marco Mene Mar 9 '16 at 8:31
  • $\begingroup$ It is ok. With PCA. Not with FA. $\endgroup$ – ttnphns Mar 9 '16 at 10:22
  • 2
    $\begingroup$ The answer to the title question: Yes it does. $\endgroup$ – amoeba Mar 9 '16 at 12:27
  • 1
    $\begingroup$ The opposite is true: if the determinant of a (correlation) matrix were not very small compared to $1$, then you would conclude there is very little redundancy among the variables--they are close to mutually orthogonal--and therefore PCA would be unlikely to reduce the dimensions. $\endgroup$ – whuber Mar 9 '16 at 15:26

Having a very small $ \det(R) $ only means that you have some variables that are almost linearly dependent. Note that $\det(R)$ equals the product of the eigenvalues of $R$; so there is at least one eigenvalue that is approximately zero.

This only means that you have some extra/redundant dimensions in your dataset and that PCA will actually be able to represent 100% of the information with a smaller ($p_\text{new} \le p - 1$) set of dimensions.

  • $\begingroup$ I slightly edited your answer to clarify it. +1. Welcome to CrossValidated! $\endgroup$ – amoeba Mar 9 '16 at 12:27
  • $\begingroup$ The conclusions of this answer are sometimes correct, but there are many other more benign and interesting possibilities. After all, the conditions in the question imply the geometric mean of the eigenvalues is a relatively high $(10^{-100})^{1/200}\approx 0.32$, so you have no basis to conclude "at least one eigenvalue ... is approximately zero." $\endgroup$ – whuber Mar 9 '16 at 15:23
  • $\begingroup$ @whuber, that is a good observation! But of course the arithmetic mean of the eigenvalues should be equal to $1$. This leads to an interesting puzzle: given that $\sum_{i=1}^{200} \lambda_i = 200$ and $\prod \lambda_i = 10^{-100}$, what is the maximum possible $\min(\lambda_i)$? $\endgroup$ – amoeba Mar 9 '16 at 21:19
  • 1
    $\begingroup$ @amoeba It's pretty straightforward, once you recognize that $199$ of the $\lambda_i$ will be the same. You only have to minimize a linear objective function subject to some linear and nonlinear constraints. I obtain $0.306704$ (which is related to the root of a degree-200 polynomial). $\endgroup$ – whuber Mar 9 '16 at 22:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.