# Where is the indeterminacy of factor values on this plot explaining factor analysis?

It is a well-known fact that in principal component analysis (PCA) we can obtain true values of components but in factor analysis (FA) we cannot obtain true values of common factors. We can compute factor scores - for example, by regression method - but these are only reasonable surrogates of the ever-unknown factor values. The same regressional method in PCA gives component scores which are exactly the true component values. Methods to compute component and factor scores are described here.

From mathematical point of view, the indeterminacy of factor values (= the inexactness of factor scores) is quite clear. The reason for it is that we don't know values of unique factors (aka factor noise) at case level. Common factor model (in toto - the system of p variables' equations) differs from the PCA model is that it is rather a data-expansion than data-reduction technique. There are assumed m common + p unique latent factors: m+p > p (whereas in PCA just m latent axes + observed "residual" or "discrepancy"). In the FA's overparameterized situation it is impossible to compute unequivocally values of m common factors out of values of p variables due to the insufficient information. We can, however, derive unequivocal statistics for the factors: covariances (correlations) between the factors and the variables: the loadings.

OK. But why do we not see the factor indeterminacy on a picture which otherwise explains FA geometrically correctly? Below is a "vector representation in subject space", copied from here. It explains the gist of FA (the explanation is here). Subject space is briefly explained in the beginning of this answer. Subject space is simply "inside-down" scatter plot with variables as points and subjects as axes, then all the many redundant dimensions are concealed.

Just briefly what's going on. 1st principal component (thin red vector) lies in the space spanned by the variables (two blue vectors), white "plane X". Factor (fat red vector) overruns that space. Factor's orthogonal projection on the plane (thin grey vector) is the regressionally estimated factor scores. By the definition of linear regression, these factor scores are the best, in terms of least squares, approximation of factor available by the variables.

Now back to our question. Configuration is fixed in the space of (hidden) N axes-individuals; indeed:

• Coordinates of $X_1$ and $X_2$ endpoints would be the N values of the two (centered) variables.
• Coordinates of the component endpoint would be the N values of the component.
• Coordinates of the factor scores endpoint would be the N scores themselves.

Likewise, coordinates of the factor $F$ endpoint should be the N true factor values. Why then would one say factor values are indeterminate? Where's your indeterminacy on the well-determined plot?

• The initial statement "It is a well-known fact that in principal component analysis (PCA) we can obtain true values of components but in factor analysis (FA) we cannot obtain true values of common factors" to me doesn't correspond to a well-known fact at all. It's a point of view that PCA is a limiting or even degenerate case of FA; it's a point of view that PCA is just a transformation procedure, so the idea of true scores or an underlying model is just foisted upon PCA from FA. It's fine to have a different view, but preference as a matter of taste, or even logic, isn't factual at all. – Nick Cox Dec 10 '14 at 13:04
• I've edited mildly on matters of English. Naturally I hope that my editing doesn't change any major or minor points in your argument: please check. – Nick Cox Dec 10 '14 at 13:10
• @ttnphns: This is exactly one of the things that has been troubling me in this exposition of yours, and I was reminded about it the other day when you brought these figures up again. I am glad that you started this discussion; maybe I will finally be able to understand how to connect your graphical explanations of FA-vs-PCA with my algebraic understanding of it. – amoeba Dec 10 '14 at 14:58
• @Nick, thank you for editing grammar. Your initial comment, however, sounds to me irrelevant. In the question, I don't raise the FA vs PCA discussion. Nor I state that PCA is a form of FA. I simply meant that: both models imply some values of a factor/component; factor values cannot be computed exactly, while component values can. It is a well-known fact. – ttnphns Dec 10 '14 at 15:23
• Sorry, but I really don't think it helps to use "model" in an undefined broad sense in statistics; it's a good word and better off not being debased. In any case, regardless of terminology, I don't see that (standard) PCA entails any discussion of errors, or indeed how that fits the mathematical definition of PCs in terms of linear algebra. Let us leave the discussion here as a flag to those who might be interested. – Nick Cox Dec 10 '14 at 15:40

My answer is: you cannot see the condition of indeterminacy of factor $F$ on the above 3D plot because you will need 4D space to see it.

Let us, for a moment, reduce the whole picture by one dimension by dropping one of two X variables, while leaving there the prerequisites of factor analysis. (Please don't take the action for re-doing FA on a single variable - it is impossible. It is just imaginary deletion of one of the variables in order to spare one dimension.) So, we have some variable $X$ (centered), in the subject space of, say, N=3 individuals. The values are the coordinates onto the individuals:

ID X
1  2
2 -2
3  0


As the things go in FA, we must decompose $X$ into $F$ the common factor, and $U$, the unique factor, both orthogonal, but neither of them coinciding or orthogonal with $X$. $F$ and $U$ will define a plane, call it "plane U". The angle between $X$ and $F$ is determined from the analysis and it gives loading $a$ - the coordinate of $X$ on $F$.

We soon discover that the solution is not unique relative axes-individuals 1,2,3. Look at the left picture. Here "plane U" (grey) is defined to coincide with horizontal plane defined by axes 1-2 (beige). It may look a bit reclinate, but it's an illusion - it is actually a bit rotated about axis 3 because angle FX is somewhat less than angle UX. Now look at the right picture. Here, clearly, "plane U" is rotated about vector $X$ to become almost perpendicular to the horizontal "plane 12". In both cases we did not alter the coordinates of $X$ onto $U$ and $F$, including loading $a$ - we only spinned arbitrarily the same plane about the straight line $X$. Thereby we changed coordinates of endpoints of $F$ and $U$ onto axes 1,2,3. The coordinates, which are the values of factor $F$ and unique factor $U$.

Thus, we've just observed the indeterminacy of factor values. Factor can be determined in FA up to its loadings and its variance only; a infinite number of solutions exist in regards to factor values, - true factor values will always remain under question.

We've shown the indeterminacy by spinning "plane U" around axis of $X$, that is, a 2D space was revolving about 1D space in 3D space. A plane can revolve about some straight line in it, in a space; a line can revolve about some point in it, in a plane; in general: a q-dimensional space can freely spin about its q-1 dimensional subspace in a q+1 dimensional superspace.

Having grasped that, let's return to the initial picture posted with the question. Bringing back the temporarily removed second X variable, we now have 2D "plane X" and consequently 3D "space U" (it consists of orthogonally intersecting planes U1 and U2). That latter may freely spin about "plane X". As it spins - without changing any of the loadings ($a$'s) or vector lengths (variances) - the endpoint of $F$ rushes within the parent space, the N-dimensional space of subjects. But to be able to show it we need 3+1= 4D space (the "q+1 dimensional superspace"), which we cant'd draw in our world. So, we can't see the indeterminacy of factor values on that (geometrically correct) 3D picture, but it is there.

What about component values/scores and factor scores? Both are computed as linear combinations of variables, and so their vectors lie in "plane X". Component scores are true component values. Factor scores are approximations of unknown true factor values. Both component and factor scores can be fully determined in the analysis. If we apply once again to the pictures of this answer, showing the "reduced one-variable example", we'll find that the component or the factor scores variate should lie within 1D space X, the $X$ itself. So, no revolving can occur. The length of the component/variate vector is defined in the analysis, and its endpoint gets fixed in 3D space of individuals. No indeterminacy.

To conclude (staring again at the initial plot): What lies in the space X of the observed variables - is fixed, up to the case values. What transcends that space - namely the m-dimensional common factor space (m=1 in our situation) + the p-dimensional unique factors space, orthogonally intersecting with the former - is freely turnable, in a lump, about the space X in the grand space of N subjects. Therefore factor values are not fixed, while component values or estimated factor scores are.

• I am confused by the following: "we must decompose X into F the common factor, and U, the unique factor, both orthogonal, but neither of them coinciding or orthogonal with X" -- why are F and U not allowed to coincide or be orthogonal to X? To me factor analysis on 1 variable seems badly under-determined: covariance matrix is $1\times 1$, so just a number $c$, and we try to fit 1 value of loading $a$ and 1 value of the noise variance (uniqueness) $\psi$. The splitting between them is arbitrary, as long as $a=c+\psi$. Why would e.g. F=X and U orthogonal to them not be a valid solution? – amoeba Dec 10 '14 at 15:09
• @amoeba, you might be correct from a broad scholastic point of view, but look - U orthogonal to X means zero uniqueness, and the case degenerates into PCA. Most programs of FA just won't allow zero uniqueness for mathematical-programming reasons. – ttnphns Dec 10 '14 at 15:15
• Well, okay. I guess my main point is that in this answer you try to argue why factor values are not uniquely determined given X and given computed [unique] loadings and [unique] uniquenesses. And your explanation might be correct (I am still thinking about it). But in the 1D case there is unfortunately another indeterminacy, because loading and uniqueness are not uniquely defined themselves! Even if you don't want to allow $\psi=0$, there is still infinite ways to split $c$ into $a+\psi$. – amoeba Dec 10 '14 at 15:19
• @amoeba, I did not attempt to do FA on a single variable. It is impossible. I just removed "temporarily" one dimension from the scene to spare room to show the revolving. It was merely a visional experiment. I hope so far that as such - it was valid. – ttnphns Dec 10 '14 at 15:28

Per Stanley Mulaik's fine text,

Mulaik, S. A. (2009). Foundations of factor analysis. Chapman and Hall/CRC.

you can imagine the many sets of possible factor scores as forming a cone. Every set of proper factor scores (that is, every set of values that includes the degree of arbitrary variance made necessary by the model) would be one vector along the outside of the cone. As factor indeterminacy is reduced (by adding more observed variable indicators with stronger loadings), the width of the cone decreases, until, at the limit, the factor becomes determinate, or nearly so. Regression method factor scores would be the line through the center of the cone. When the factor becomes determinate / a composite, then regression method scores would coincide with actual factor scores.