Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

SPSS offers several methods of factor extraction:

  1. Principal components (which isn't factor analysis at all)
  2. Unweighted least squares
  3. Generalized least squares
  4. Maximum Likelihood
  5. Principal Axis
  6. Alpha factoring
  7. Image factoring

Ignoring the first method, which isn't factor analysis (but principal component analysis, PCA), which of these methods is "the best"? What are the relative advantages of the different methods? And basically, how would I choose which one to use?

Additional question: should one obtain similar results from all 6 methods?

share|improve this question
    
Hmm, my first impulse: isn't there a wikipedia entry on this? If not - surely there should exist one... – Gottfried Helms Feb 24 '13 at 14:49
    
I have recently been doing an EFA using tetrachorics - resulted in non-positive definite correlation matrix - so required smoothing - but still ill conditioned. ML gave quite different loadings to WLS and ULS, ULS gives incorrect standard error, so used a WLS. In short i think the best method will depend on the data – user20650 Feb 24 '13 at 15:24
3  
Yes, there is a wikipedia article. It says to use MLE if the data are normal and PAF otherwise. It doesn't say much about the merits or otherwise of the other options. In any case, I would be happy to know what the members of this site think about this issue, based on their practical experience. – Placidia Feb 24 '13 at 15:44
up vote 22 down vote accepted

To make it short. Two last methods are each very special and are different from numbers 2-5 which are all called common factor analysis and are indeed seen as alternatives. And, most of the time, they give rather similar results. They are "common" because they represent classical factor model, the common factors + unique factors model. It is this model which is typically used in questionnaire analysis/validation.

Principal Axis (PAF), aka Principal Factor with iterations is the oldest and perhaps yet quite popular method. It is iterative PCA$^1$ application to the matrix where communalities stand on the diagonal in place of 1s. Each next iteration thus refines communalities further until they converge. In doing so, the method that seeks to explain variance, not pairwise correlations, eventially explains the correlations. Principal Axis method has the advantage in that it can, like PCA, analyze not only correlations, but also covariances and other SSCP measures (raw sscp, cosines). The rest three methods process only correlations [in SPSS; covariances could be analyzed in some other implementations]. This method is dependent on the quality of starting estimates of communalities (and it is its disadvantage). Usually the squared multiple correlation/covariance is used as the starting value, but you may prefer other estimates (including those taken from previous research). Please read this for more. If you want to see an example of Principal axis factoring computations, commented and compared with PCA computations, please look in here.

Ordinary or Unweighted least squares (ULS) is the algorithm that directly aims at minimizing the residuals between the input correlation matrix and the reproduced (by the factors) correlation matrix (while diagonal elements as the sums of communality and uniqueness are aimed to restore 1s). This is the straight task of FA$^2$. ULS method can work with singular and even not positive semidefinite matrix of correlations provided the number of factors is less than its rank, - although it is questionnable if theoretically FA is appropriate then.

Generalized or Weighted least squares (GLS) is a modification of the previous one. When minimizing the residuals, it weights correlation coefficients differentially: correlations between variables with high uniqness (at the current iteration) are given less weight$^3$. Use this method if you want your factors to fit highly unique variables (i.e. those weakly driven by the factors) worse than highly common variables (i.e. strongly driven by the factors). This wish is not uncommon, especially in questionnaire construction process (at least I think so), so this property is advantageous$^4$.

Maximum Likelihood (ML) assumes data (the correlations) came from population having multivariate normal distribution (other methods make no such an assumption) and hence the residuals of correlation coefficients must be normally distributed around 0. The loadings are iteratively estimated by ML approach under the above assumption. The treatment of correlations is weighted by uniqness in the same fashion as in Generalized least squares method. While other methods just analyze the sample as it is, ML method allows some inference about the population, a number of fit indices and confidence intervals are usually computed along with it [unfortunately, mostly not in SPSS, although people wrote macros for SPSS that do it]. The general fit chi-square test asks if the factor-reproduced correlation matrix can pretend to be the population matrix of which the observed matrix is random sampled.

All the methods I briefly described are linear, continuous latent model. "Linear" implies that rank correlations, for example, should not be analyzed. "Continuous" implies that binary data, for example, should not be analyzed (IRT or FA based on tetrachoric correlations would be more appropriate).


$^1$ Because correlation (or covariance) matrix $\bf R$, - after initial communalities were placed on its diagonal, will usually have some negative eigenvalues, these are to be kept clean of; therefore PCA should be done by eigen-decomposition, not SVD.

$^2$ ULS method includes iterative eigendecomposition of the reduced correlation matrix, like PAF, but within a more complex, Newton-Raphson optimization procedure aiming to find unique variances ($\bf u^2$, uniquenesses) at which the correlations are reconstructed maximally. In doing so ULS appears equivalent to method called MINRES (only loadings extracted appear somewhat orthogonally rotated in comparison with MINRES) which is known to directly minimize the sum of squared residuals of correlations.

$^3$ GLS and ML algorithms are basically as ULS, but eigendecomposition on iterations is performed on matrix $\bf uR^{-1}u$ (or on $\bf u^{-1}Ru^{-1}$), to incorporate uniquenesses as weights. ML differs from GLS in adopting the knowledge of eigenvalue trend expected under normal distribution.

$^4$ The fact that correlations produced by less common variables are permitted to be fitted worse may (I surmise so) give some room for the presence of partial correlations (which need not be explained), what seems nice. Pure common factor model "expects" no partial correlations, which is not very realistic.

enter image description here

share|improve this answer
    
I think, one should add more aspect: whether we use the methods to fit a factor solution to a prespecified number of factors, or whether the number of factors should emerge from the data, by some criterion (eigenvalue, screetest,...). As I understand, ML is only sensical if you prespecify a number of factors, and then a factor solution will be sought, and even a chi-square-test is then possible. PCA lets the number of factors dynamically appear by data-properties, given some criterion, no chi-square test. PAF can be used in both ways. – Gottfried Helms Feb 25 '13 at 8:15
1  
@Gottfried, I'd rather disagree with the way you put it. All FA methods require the number of factors m be known: you fit the model for m you specify. You can use various criterions which may help to decide on m, but all these are not part of factor extraction methods themselves. With the exception of that chi-square, computed along with GLS and ML methods. Also, with PA method, if you know true communalities in advance (which is very seldom), you can make them to guide you towards the best m. But in any case, its you, not an extraction algorithm, decides on m. – ttnphns Feb 25 '13 at 11:57
    
Now what should we use? Which one is the best? – MrYouMath Jun 18 at 17:37
    
The best is what you like best. You choose, then if needed you explain why it suits you. As everywhere. – ttnphns Jun 18 at 17:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.