Best factor extraction methods, with reference to SPSS

SPSS offers several methods of factor extraction:

1. Principal components (which isn't factor analysis at all)
2. Unweighted least squares
3. Generalized least squares
4. Maximum Likelihood
5. Principal Axis
6. Alpha factoring
7. Image factoring

Ignoring the first method, which isn't factor analysis, which of these methods is "the best"? what are the relative advantages of the different methods? And basically, how would I choose which one to use?

Similar question: should one obtain similar results from all 6 methods?

-
Hmm, my first impulse: isn't there a wikipedia entry on this? If not - surely there should exist one... –  Gottfried Helms Feb 24 '13 at 14:49
I have recently been doing an EFA using tetrachorics - resulted in non-positive definite correlation matrix - so required smoothing - but still ill conditioned. ML gave quite different loadings to WLS and ULS, ULS gives incorrect standard error, so used a WLS. In short i think the best method will depend on the data –  user20650 Feb 24 '13 at 15:24
Yes, there is a wikipedia article. It says to use MLE if the data are normal and PAF otherwise. It doesn't say much about the merits or otherwise of the other options. In any case, I would be happy to know what the members of this site think about this issue, based on their practical experience. –  Placidia Feb 24 '13 at 15:44

To make it short. Two last methods are each very special and are different from numbers 2-5 which are all called common factor analysis and are indeed seen as alternatives. And, most of the time, they give rather similar results. They are "common" because they represent classical factor model, the common factors + unique factors model. It is this model which is typically used in questionnaire analysis/validation.

Principal Axis (= Principal Factor with iterations) is the oldest and perhaps yet most popular method. It is iterative PCA$^1$ application to the matrix where communalities stand on the diagonal in place of 1s. Each next iteration thus refines communalities further until they converge. In doing so, the method that seeks to explain variance, not pairwise correlations, eventially explains the correlations. Principal Axis method has the advantage in that it can, like PCA, analyse not only correlations, but also covariances and other SSCP measures (raw sscp, cosines). The rest three methods process only correlations [in SPSS; covariances could be analyzed in some other implementations]. This method is dependent on the quality of starting estimates of communalities (and it is its disadvantage). Usually the squared multiple correlation/covariance is used as the starting value, but you may prefer other estimates (including those taken from previous research). Please read this for more. If you want to see an example of Principal axis factoring computations, commented and compared with PCA computations, please look in here.

Ordinary or Unweighted least squares is the algorithm that directly aims at minimizing the residuals between the input correlation matrix and the reproduced (by the factors) correlation matrix (while diagonal elements as the sums of communality and uniqueness are aimed to restore 1s). This is the straight task of FA. The algorithm resembles a bit those used in MDS (e.g. ALSCAL), actually it uses Newton-Raphson algorithm occupied with computation of derivative functions.

Generalized or Weighted least squares is a modification of the previous one. When minimizing the residuals, it weights correlation coefficients differentially: correlations between variables with high uniqness (at the current iteration) are given less weight. Use this method if you want your factors to fit highly unique variables (i.e. those weakly driven by the factors) worse than highly common variables (i.e. strongly driven by the factors). This wish is not uncommon, especially in questionnaire construction process (at least I think so), so this property is advantageous$^2$.

Maximum Likelihood assumes data (the correlations) came from population having multivariate normal distribution (other methods make no such an assumption) and hence the residuals of correlation coefficients must be normally distributed around 0. The loadings are iteratively estimated by ML approach under the above assumption. The treatment of correlations is weighted by uniqness in the same fashion as in Generalized least squares method. While other methods just analyze the sample as it is, ML method allows some inference about the population, a number of fit indices and confidence intervals are usually computed along with it [unfortunately, mostly not in SPSS, although people wrote macros for SPSS that do it].

All the methods I briefly described are linear, continuous latent model. "Linear" implies that rank correlations, for example, should not be analyzed. "Continuous" implies that binary data, for example, should not be analyzed (IRT or FA based on tetrachoric correlations would be more appropriate).

$1$ Because matrix with communalities on the diagonal usually will have some negative eigenvalues, these are to be kept clean of; therefore PCA should be done by eigen-decomposition, not SVD.

$2$ The fact that correlations produced by less common variables are permitted to be fitted worse may (I think so) give some room for the presence of partial correlations (which need not be explained), what seems nice. Pure common factor model "expects" no partial correlations, which is not very realistic.

-
I think, one should add more aspect: whether we use the methods to fit a factor solution to a prespecified number of factors, or whether the number of factors should emerge from the data, by some criterion (eigenvalue, screetest,...). As I understand, ML is only sensical if you prespecify a number of factors, and then a factor solution will be sought, and even a chi-square-test is then possible. PCA lets the number of factors dynamically appear by data-properties, given some criterion, no chi-square test. PAF can be used in both ways. –  Gottfried Helms Feb 25 '13 at 8:15
@Gottfried, I'd rather disagree with the way you put it. All FA methods require the number of factors m be known: you fit the model for m you specify. You can use various criterions which may help to decide on m, but all these are not part of factor extraction methods themselves. With the exception of that chi-square, computed along with GLS and ML methods. Also, with PA method, if you know true communalities in advance (which is very seldom), you can make them to guide you towards the best m. But in any case, its you, not an extraction algorithm, decides on m. –  ttnphns Feb 25 '13 at 11:57