7
$\begingroup$

If $X$ is a $m × n$ matrix, where $m$ is the number of measurement types (variables) and $n$ is the number of samples, would it be correct to perform a PCA on a matrix that has $m \geq n$ ? If not, please provide some arguments why would this be a problem.

I remember having heard that doing such an analysis would be invalid, but the Wikipedia page for PCA doesn't mention a low $n/m$ ratio as being a potential limitation for using the method.

Please note that I am a biologist and aim at a more practical answer (if possible).

$\endgroup$
3
$\begingroup$

PCA of variables. Number of observations n is low relative to number of variables. 1) Mathematical aspect. Whenever n<=m correlation matrix is singular which means some of last m principle components are zero-variance, that is, they are not existant. This is not a problem to PCA, generally speaking, since you could just ignore those. However, many software (mostly those uniting PCA and Factor Analysis in one command or procedure) will not allow you to have singular correlation matrix. 2) Statistical aspect. To have your results reliable you must have correlations reliable; that requires considerable sample size which always should be larger than number of variables. They say, if you have m=20 you ought to have n=100 or so. But if you have m=100 you should have n=300 or so. As m grows, minimal recommended n/m proportion diminishes.

$\endgroup$
2
  • 1
    $\begingroup$ Thank you for the answer! Having mentioned the correlation matrix, now the n/m problem is clearer, but how do you define a "reliable result"? If the PCA has been performed in order to simplify and inspect the data structure, how can one notice that the results are not reliable? $\endgroup$
    – ils
    Mar 14 '11 at 13:01
  • 1
    $\begingroup$ Mentioning "relible" simply nods towards "representative sample" theme. Mathematically one has error-free regression line with 2 variables and 2 points (individuals). But what if one of the latter two is outlier. No, let us have as many individuals as we can collect, to balance atypical deviates. With your correlation matrix, you just will want to be sure that high correlations in it are significant (reliable). $\endgroup$
    – ttnphns
    Mar 14 '11 at 13:26
3
$\begingroup$

Matrix dimension by itself have little todo with PCA validity. What will change is the interpretation of your data and it's all dependent on how you want to use the result.

PCA is very powerful to use to find anomalys or outliers in your data. Maybe you have performed an experiment on two different days, used different machines in the experiment etc. If the purpose is to get an overview of data PCA is one of the most efficient way to do that regardless of any n/m ratios.

If your main interest is to investigate clusters or relations between samples, then #variables are not very important. (But other type of statistics on the result might be important if #samples are low).

If you look at individual variables, then they will be less reliable if you have few samples. However, this is a problem you will have with any other method as well. If you find patterns among variables that make sense then you should certainly not disregard your findings because you have a low n/m ratio. However, few observations are almost allways problematic and should lead to caution in interpretation and the more samples you have the less important is the #sample/#variable relation.

$\endgroup$
2
$\begingroup$

I do not think that you would get any useful information from such an analysis, as lore in my subject area (psychology) suggests a 10;1 ratio in favour of n as a precondition. In some circumstances (where communalities are high) you can get away with 5 or 3 to 1, but a ratio of less than 1 is probably a recipe for disaster.

$\endgroup$
3
  • $\begingroup$ Thank you very much for the quick response! Could you briefly elaborate what the consequences of such an "disaster" would be? A PCA could be done on such a matrix anyway, some hints about the potential problems will be welcome. $\endgroup$
    – ils
    Mar 14 '11 at 12:50
  • $\begingroup$ As I have already received some hints from @ttnphns concerning the reliability problems that might occur, could you point me at some references that treat the problem and suggest the ration you mentioned? $\endgroup$
    – ils
    Mar 14 '11 at 14:07
  • 2
    $\begingroup$ @ils well, i know that R will not allow you to fit this model. I got the numbers from Tabachnick & Fidell, Understanding Multivariate Statistics, and a closer look at these assumptions from MacCallum et al notendur.hi.is/adg11/Proffraedi/… $\endgroup$ Mar 14 '11 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.