Factor extraction methods: Harris and Image extraction

Question

Is Harris's (1962) extraction method considered a form of principal component analysis (PCA) or factor analysis (FA)? What about image extraction?

I am using SAS and the documentation says:

HARRIS | H

yields Harris component analysis of $\mathbf{S}^{-1}\mathbf{RS}^{-1}$ (Harris 1962), a noniterative approximation to canonical component analysis.

IMAGE | I

yields principal component analysis of the image covariance matrix, not the image analysis of Kaiser (1963, 1970) or Kaiser and Rice (1974). A nonsingular correlation matrix is required.

I also found an additional reference in an outdated SAS technical manual that says:

The problem of factor score indeterminacy has led several factor analysts to propose methods yielding components that can be considered approximations to common factors. Since these components are defined as linear combinations, they are computable. The methods include Harris component analysis and image component analysis. The advantage of producing determinate component scores is offset by the fact that, even if the data fit the common factor model perfectly, component methods do not generally recover the correct factor solution. You should not use any type of component analysis if you really want a common factor analysis (Dziuban and Harris 1973; Lee and Comrey 1979).

Based on the labeling of the methods in the documentation and the description in the tech manual, I would be inclined to assume they are PCA methods. However, after doing some reading it appears that both methods attempt to decompose the shared or common variance (a distinguishing characteristic of FA).

I am currently not concerned about picking the best method but just want to understand the roots of these two methods. Unfortunately I do not have access to the journals where the original papers about the methods were published.

Answer 1

[@ttnphns 18.03.2017 note: this answer as currently may contain some imprecision, to revise soon].

Neither Guttman's Image Analysis nor the Harris (1962) account of it are true factor analysis.

Classic or Common factor FA model of restoration the correlation or covariance matrix $\bf R$, usually called "fundamental factor theorem" in the literature of FA, looks like

$\bf R \approx AA' + U^2$, where $\bf A$ is the loadings of $m$ extracted orthogonal factors and $\bf U^2$ is the diagonal matrix of uniquenesses (the portions of variable variances unexplained by the common factors; diagonal of $\bf R$ minus the uniquenesses are called communalities, the explained portion).

The problem of the model is that, although most of the time uniquenesses (or communalities) can be estimated practically on iterations of factor extraction algorithms, these quantities are not mathematically defined, in any closed form, beforehand; common factor analysis hunts for them blindly by trial, as if for some ghosts. And common FA, even having obtaned their estimates in the end, still does not know the values of the uniquenesses on case level, which is the cause why factor values are not determinable precisely.

Guttman was dissatisfied with that, he wanted the factors and their scores be well-defined and computable from the data unequivocally. He put forward Image analysis as an alternative to common factor analysis. Its fundamental theorem is

$\bf R \approx D-E+2S^2$, where $\bf D$ is the covariance image and $\bf E$ is the covariance antimage matrices and $\bf S^2$ is the diagonal of the latter one. (These matrices are often used before common factor analysis to estimate initial values of communalities and do some diagnostics.) Image of a variable, a diagonal value of $\bf D$, is its variance multiplied by the R-square of its dependency on all the other variables; and anti-image of it is the variance minus the image. Guttman defined image to be in place of the communality and anti-image to be in place of the uniqueness.

When $m=p$, the number of all the variables, this is precise equation. Image analysis as the substitute for Factor analysis amounts to the eigendecomposition of $\bf D$ - the matrix which replaces here the reduced $\bf R$ (i.e. $\bf R$ with initial estimates of communalities on its diagonal) of common FA. Then the first strong $m$ principal axes of $\bf D$ can be seen as "common" factors; you compute their loadings $\bf A$ as done in PCA.

There are practical problems observed in defining an optimal $m$ with the eigenvalues of $\bf D$ (without iterations, or you'll fall back into factor analysis "underdefinedness"). Harris' (1962) suggestions on this topic are considered most correct (better than earlier Kaiser's), and they, by the way, links image analysis to a more sophisticated non common-factor model FA, Canonical factor analysis. I can't go further and explain Harris proposals and canonical FA because I know it not well.

Guttman's Image analysis is not factor analysis in true sense because the off-diagonal elements of $\bf E$ are not zero: anti-images correlate with each other; they also are not orthogonal to the "common" factors produced by the decomposition of $\bf D$. Image analysis can be thought of as a specially weighted principal component analysis being halfway between PCA and the true common factor analysis. Image analysis is rarely used nowadays because the mentioned Guttman's demands for close-form definitions are being considered too positivistic.

So, in my inderstanding, Image analysis is technically a PCA but is performed on a matrix which is more "saturated" with communality than $\bf R$ but less than the reduced $\bf R$ which is estimated and re-estimated on the iterations of the true factor analysis. Harris' proposals - to my little awareness - are the approach to select the best $m$ in image analysis, partly based on canonical correlations viewpoint.

---

One should not confuse Guttman's Image analysis with Jöreskog's Image Factor analysis having the fundamental theorem formula

$\mathbf {R \approx AA'} + k \bf S^2$, approximating the image analysis idea closer to the common-factor model. Since the image of a variable is known to be the lower bound of its true communality, the uniqueness is consequently the anti-image multiplied by some coefficient $k$ in the range between 0 and 1: $u_i^2=k_is_i^2$. Of course, $k$ is different for each variable i; but Jöreskog proposed to assume constant $k$ approximating the anti-images to the uniquenesses, $\mathbf U^2 \approx k \bf S^2$.

As the number $p$ of variables increase that model becomes very good approximation of the common factor analysis. The algorithm (which is noniterative) is as follows. Having anti-images $\bf S^2$, matrix $\bf M=S^{-1}RS^{-1}$ is computed and eigendecomposed. The approach to discard $p-m$ last nonsignificant eigenvalues of it is analogous to how it is done in Bartlett's test of significant principal components (see e.g. here) which is performed on $\bf R$: but here $\bf M$ is instead of it. The test will allow you to choose number of factors $m$; and $k$ being the mean of the $p-m$ last eigenvalues. Having estimated the two values, compute the factor loadings as $\bf A=SVH$, where $\bf V$ is the first $m$ eigenvectors of $\bf M$ and $\bf H$ is the diagonal matrix of the first $m$ eigenvalues of it with $k$ subtracted from each and then sq. root taken.

Image factor analysis is thus an improvement of Image analysis making it more in line with the standard FA model.