8
$\begingroup$

Some scientific papers report results of parallel analysis of principal axis factor analysis in a way inconsistent with my understanding of the methodology. What am I missing? Am I wrong or are they.

Example:

  • Data: The performance of 200 individual humans has been observed on 10 tasks. For each individual and each task, one has a performance score. The question now is to determine how many factors are the cause for the performance on the 10 tasks.
  • Method: parallel analysis to determine the number of factors to retain in a principal axis factor analysis.
  • Example for reported result: “parallel analysis suggests that only factors with eigenvalue of 2.21 or more should be retained”

That is nonsense, isn’t it?

From the original paper by Horn (1965) and tutorials like Hayton et al. (2004) I understand that parallel analysis is an adaptation of the Kaiser criterion (eigenvalue > 1) based on random data. However, the adaptation is not to replace the cut-off 1 by another fixed number but an individual cut-off value for each factor (and dependent on the size of the data set, i.e. 200 times 10 scores). Looking at the examples by Horn (1965) and Hayton et al. (2004) and the output of R functions fa.parallel in the psych package and parallel in the nFactors package, I see that parallel analysis produces a downward sloping curve in the Scree plot to compare to the eigenvalues of the real data. More like “Retain the first factor if its eigenvalue is > 2.21; additionally retain the second if its eigenvalue is > 1.65; …”.

Is there any sensible setting, any school of thought, or any methodology that would render “parallel analysis suggests that only factors with eigenvalue of 2.21 or more should be retained” correct?

References:

Hayton, J.C., Allen, D.G., Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: a tutorial on parallel analysis. Organizational Research Methods, 7(2):191-205.

Horn, J.L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2):179-185.

$\endgroup$
  • 1
    $\begingroup$ Incidentally, Hayton et al.'s requisite that the distributional form of the uncorrelated data used to generate mean eigenvalues to estimate "sampling bias" was critically examined and rejected in Dinno, A. (2009). Exploring the Sensitivity of Horn’s Parallel Analysis to the Distributional Form of Simulated Data. Multivariate Behavioral Research, 44(3):362–388. $\endgroup$ – Alexis May 15 '14 at 15:15
  • 1
    $\begingroup$ Also, incidentally see my parallel analysis package paran for R (on CRAN) and for Stata (within Stata type findit paran). $\endgroup$ – Alexis May 15 '14 at 21:41
8
$\begingroup$

There are two equivalent ways to express the parallel analysis criterion. But first I need to take care of a misunderstanding prevalent in the literature.

The Misunderstanding
The so-called Kaiser rule (Kaiser didn't actually like the rule if you read his 1960 paper) eigenvalues greater than one are retained for principal component analysis. Using the so-called Kaiser rule eigenvalues greater than zero are retained for principal factor analysis/common factor anlaysis. This confusion has arisen over the years because several authors have been sloppy about using the label "factor analysis" to describe "principal component analysis," when they are not the same thing.

See Gently Clarifying the Application of Horn’s Parallel Analysis to Principal Component Analysis Versus Factor Analysis for the math of it if you need convincing on this point.

Parallel Analysis Retention Criteria
For principal component analysis based on the correlation matrix of $p$ number of variables, you have several quantities. First you have the observed eigenvalues from an eigendecomposition of the correlation matrix of your data, $\lambda_{1}, \dots, \lambda_{p}$. Second, you have the mean eigenvalues from eigendecompositions of the correlation matrices of "a large number" of random (uncorrelated) data sets of the same $n$ and $p$ as your own, $\bar{\lambda}^{\text{r}}_{1},\dots,\bar{\lambda}^{\text{r}}_{p}$.

Horn also frames his examples in terms of "sampling bias" and estimates this bias for the $q^{\text{th}}$ eigenvalue (for principal component analysis) as $\varepsilon_{q} = \bar{\lambda}^{\text{r}}_{q} - 1$. This bias can then be used to adjust observed eigenvalues thus: $\lambda^{\text{adj}}_{q} = \lambda_{q} - \varepsilon_{q}$

Given these quantities you can express the retention criterion for the $q^{\text{th}}$ observed eigenvalue of a principal component parallel analysis in two mathematically equivalent ways:

$\lambda^{\text{adj}}_{q} \left\{\begin{array}{cc} > 1 & \text{Retain.} \\\\ \le 1 & \text{Not retain.} \end{array}\right.$

$\lambda_{q} \left\{\begin{array}{cc} > \bar{\lambda}^{\text{r}}_{q} & \text{Retain.} \\\\ \le \bar{\lambda}^{\text{r}}_{q} & \text{Not retain.} \end{array}\right.$

What about for principal factor analysis/common factor analysis? Here we have to bear in mind that the bias is the corresponding mean eigenvalue: $\varepsilon_{q} = \bar{\lambda}^{\text{r}}_{q} - 0 = \bar{\lambda}^{\text{r}}_{q}$ (minus zero because the Kaiser rule for eigendecomposition of the correlation matrix with the diagonal replaced by the communalities is to retain eigenvalues greater than zero). Therefore here $\lambda^{\text{adj}}_{q} = \lambda_{q} - \bar{\lambda}^{\text{r}}_{q}$.

Therefore the retention criteria for principal factor analysis/common factor analysis ought be expressed as:

$\lambda^{\text{adj}}_{q} \left\{\begin{array}{cc} > 0 & \text{Retain.} \\\\ \le 0 & \text{Not retain.} \end{array}\right.$

$\lambda_{q} \left\{\begin{array}{cc} > \bar{\lambda}^{\text{r}}_{q} & \text{Retain.} \\\\ \le \bar{\lambda}^{\text{r}}_{q} & \text{Not retain.} \end{array}\right.$

Notice that the second form of expressing the retention criterion is consistent for both principal component analysis and common factor analysis (i.e. because the definition of $\lambda^{\text{adj}}_{q}$ changes depending on components/factors, but the second form of retention criterion is not expressed in terms of $\lambda^{\text{adj}}_{q}$).

one more thing...
Both principal component analysis and principal factor analysis/common factor analysis can be based on the covariance matrix rather than the correlation matrix. Because this changes the assumptions/definitions about the total and common variance, only the second forms of the retention criterion ought to be used when basing one's analysis on the covariance matrix.

$\endgroup$
  • 1
    $\begingroup$ Great! The first important part to me is that your retention criteria use $\bar{\lambda}^{r}_{q}$, i.e. a specific cut-off value for each factor $q$. The questionable sentence “parallel analysis suggests that only factors with eigenvalue of 2.21 or more should be retained” equals $\forall~q~\bar{\lambda}^{r}_{q}=2.21$. This is impossible. For principal component annalysis eigenvectors add up to $p$, for factor analysis to $< p$. One single $\bar{\lambda}^{r}$ irrespective of $q$ exists only for fully uncorrelated data ($n \rightarrow \infty$) and then it is either 0 (fa) or 1 (pca). Correct? $\endgroup$ – jhg May 15 '14 at 20:32
  • $\begingroup$ I had read your paper "Gently Clarifying ..." before and like it very much. In this post you state "using the so-called Kaiser rule eigenvalues greater than zero are retained for principal factor analysis/common factor anlaysis" and in the paper there is a similar comment. From the math, it is intuitive and makes total sense -- I wonder why I didn't come across this before. Are there other papers/books about this, or is "Gently Clarifying ..." the first to gently clarify that zero is the appropriate reference for principal factor analysis (if one uses the Kaiser criterion at all)? $\endgroup$ – jhg May 15 '14 at 20:40
  • 2
    $\begingroup$ Possibly: they may simply have meant that the smallest of the observed eigenvalues greater than $\bar{\lambda}^{\text{r}}$ (i.e. the ones they retained) was 2.21. There is one caveat I would add: the first form of the retention criterion has to be revised when using the covariance matrix, $\mathbf{\Sigma}$. The assumption when using $\mathbf{\Sigma}$ is that total variance (PCA) equals the sum of the observed variances of the data, and the $>1$ translates to $ > \text{trace}(\mathbf{\Sigma})/p$: this number might well be 2.21. $\endgroup$ – Alexis May 15 '14 at 20:42
  • $\begingroup$ @jhg Kaiser wrote "[Guttman's] universally strongest lower bound requires that we find the number of positive latent roots of the observed correlation matrix with squared multiples in the diagonal." But Guttman was also writing about the correlation matrix when describing unity as the critical bound of the eigenvalues of R (not R-uniquenesses) (bottom of page 154 to top of page 155), although he does not explicitly draw out the logic for R-Uniquenesses, he waves at it earlier in the middle of page 150. $\endgroup$ – Alexis May 15 '14 at 21:35
4
$\begingroup$

Yes, it is possible to have a value of 2.21 if the sample size is not infinitely large (or large enough...). This is, in fact the motivation behind the development of Parallel Analysis as an augmentation to the eigenvalue 1 rule.

I cite Valle 1999 on this answer and have italicized the part speaking directly to your question.

Selection of the Number of Principal Components:  The Variance of the Reconstruction Error Criterion with a Comparison to Other Methods† Sergio Valle,Weihua Li, and, and S. Joe Qin* Industrial & Engineering Chemistry Research 1999 38 (11), 4389-4401

Parallel Analysis. The PA method basically builds PCA models for two matrices: one is the original data matrix and the other is an uncorrelated data matrix with the same size as the original matrix. This method was developed originally by Horn to enhance the performance of the Scree test. When the eigenvalues for each matrix are plotted in the same figure, all the values above the intersection represent the process information and the values under the intersection are considered noise. Because of this intersection, the parallel analysis method is not ambiguous in the selection of the number of PCs. For a large number of samples, the eigenvalues for a correlation matrix of uncorrelated variables are 1. In this case, the PA method is identical to the AE method. However, when the samples are generated with a finite number of samples, the initial eigenvalues exceed 1, while the final eigenvalues are under 1. That is why Horn suggested comparing the correlation matrix eigenvalues for uncorrelated variables with those of a real data matrix based on the same sample size.

$\endgroup$
  • $\begingroup$ The question is whether a sole value of 2.21 can be reasonable. As the italic part in your quote from Valle et al. shows with a finite number of observations, there will (to my understanding) always be a series of decreasing eigenvalues. Thus, for each factor from the original data, there is a different eigenvalue from the parallel analysis to compare. When sample size becomes large (couple of thousand individuals), eigenvalues converge to 1. In that case I could understand one single comparison, but only at the level 1. $\endgroup$ – jhg May 15 '14 at 9:24
  • $\begingroup$ Doesn't the 2.21 here mean for this dataset and the method used (so that combination) 2.21 is the cut-off below which the eigenvalue is too small? I am not sure what you mean by "sole value." Do you mean as a general rule, like the eigenvalue 1 rule? The cutoff is different for each parallel analysis typically. $\endgroup$ – Deathkill14 May 15 '14 at 9:28
  • $\begingroup$ I understand that the parallel analysis depends on the number of variables (in my example above "10 tasks") and the number of observations (200 in the example). Thus, it is very specific for an individual dataset and there can't be a general rule like "don't use eigenvalue 1, use eigenvalue 2.21". That would be nonsense for sure. But for a specific example with 200 observations on 10 variables and, thus, 1 to 10 factors. Can it be that a parallel analysis suggests to retain factors with eigenvalue greater 2.21 independent of whether the factor is the first, second, third,...? $\endgroup$ – jhg May 15 '14 at 9:51
  • $\begingroup$ The idea of the cut-off value (say 1 or 2.21) is that below that value the variation in a factor is essentially noise (essentially noise since that is the baseline eigenvalue from the random matrix). Typically, factors are sorted from highest to lowest eigenvalue, but that is perhaps important mostly for interpretability. So "first second third" are not necessarily fixed in stone. In any case, the factors with eigenvalues greater than 2.21 in your case are assumed to contain more info than noise. Keep them. $\endgroup$ – Deathkill14 May 15 '14 at 9:59
2
$\begingroup$

Your example is certainly not clear, but it might not be nonsense either. Briefly, consider the possibility that the example is basing its decision rule on the eigenvalue of the first simulated factor that is larger than the real factor of the same factor number. Here's another example in :

d8a=data.frame(y=rbinom(99,1,.5),x=c(rnorm(50),rep(0,49)),z=rep(c(1,0),c(50,49)))
require(psych);fa.parallel(d8a)

The data are random, and there are only three variables, so a second factor certainly wouldn't make sense, and that's what the parallel analysis indicates.* The results also corroborate what @Alexis said regarding "The Misunderstanding".

Say I interpret this analysis as follows: “Parallel analysis suggests that only factors [not components] with eigenvalue of 1.2E-6 or more should be retained.” This makes a certain amount of sense because that's the value of the first simulated eigenvalue that is larger than the "real" eigenvalue, and all eigenvalues thereafter necessarily decrease. It's an awkward way to report that result, but it's at least consistent with the reasoning that one should look very skeptically at any factors (or components) with eigenvalues that aren't much larger than the corresponding eigenvalues from simulated, uncorrelated data. This should be the case consistently after the first instance on the scree plot where the simulated eigenvalue exceeds the corresponding, real eigenvalue. In the above example, the simulated third factor is very slightly smaller than the "real" third factor, but nobody in their right mind is going to retain a three-factor solution here.


*In this case, R says, "Parallel analysis suggests that the number of factors = 1 and the number of components = 2," but hopefully most of us know not to trust our software to interpret our plots for us...I definitely would not retain the second component just because it's infinitesimally larger than the second simulated component.

$\endgroup$
  • 1
    $\begingroup$ Great, creative idea how to interpret the sentence. I considered it more than briefly. It's not the case. $\endgroup$ – jhg May 15 '14 at 21:28
  • $\begingroup$ Oy. Sounds like a weird article(s) you're working with... $\endgroup$ – Nick Stauner May 15 '14 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.