It is a well-known fact that in principal component analysis (PCA) we can obtain true values of components but in factor analysis (FA) we cannot obtain true values of common factors. We can compute factor scores - for example, by regression method - but these are only reasonable surrogates of the ever-unknown factor values. The same regressional method in PCA gives component scores which are exactly the true component values. Methods to compute component and factor scores are described here.
From mathematical point of view, the indeterminacy of factor values (= the inexactness of factor scores) is quite clear. The reason for it is that we don't know values of unique factors (aka factor noise) at case level. Common factor model (in toto - the system of
p variables' equations) differs from the PCA model is that it is rather a data-expansion than data-reduction technique. There are assumed
m common +
p unique latent factors:
m+p > p (whereas in PCA just
m latent axes + observed "residual" or "discrepancy"). In the FA's overparameterized situation it is impossible to compute unequivocally values of
m common factors out of values of
p variables due to the insufficient information. We can, however, derive unequivocal statistics for the factors: covariances (correlations) between the factors and the variables: the loadings.
OK. But why do we not see the factor indeterminacy on a picture which otherwise explains FA geometrically correctly? Below is a "vector representation in subject space", copied from here. It explains the gist of FA (the explanation is here). Subject space is briefly explained in the beginning of this answer. Subject space is simply "inside-down" scatter plot with variables as points and subjects as axes, then all the many redundant dimensions are concealed.
Just briefly what's going on. 1st principal component (thin red vector) lies in the space spanned by the variables (two blue vectors), white "plane X". Factor (fat red vector) overruns that space. Factor's orthogonal projection on the plane (thin grey vector) is the regressionally estimated factor scores. By the definition of linear regression, these factor scores are the best, in terms of least squares, approximation of factor available by the variables.
Now back to our question. Configuration is fixed in the space of (hidden)
N axes-individuals; indeed:
- Coordinates of $X_1$ and $X_2$ endpoints would be the
Nvalues of the two (centered) variables.
- Coordinates of the component endpoint would be the
Nvalues of the component.
- Coordinates of the factor scores endpoint would be the
Likewise, coordinates of the factor $F$ endpoint should be the
N true factor values. Why then would one say factor values are indeterminate? Where's your indeterminacy on the well-determined plot?