Factor analysis is a dimensionality reduction latent variable technique which replaces inter-correlating variables with a smaller number of continuous latent variables called factors. The factors are believed to be responsible for the inter-correlations. [For confirmatory factor analysis, please use the tag 'confirmatory-factor'. Also, the term "factor" of factor analysis should not be confused with "factor" as categorical predictor of a regression/ANOVA.]
A latent factor (matrix) model has the following form:
$${\bf Y} = {\bf \mathbb{1}\mu}^\top + {\bf ZA}^\top + {\bf E\psi}^{1/2}$$
Where:
- ${\bf Y} \in \mathbb{R}^{n \times p}$ whose rows are random vectors in $\mathbb{R}^p$
- ${\bf \mu} \in \mathbb{R}^p$ is a column vector of means of a generic row of ${\bf Y}$
- ${\bf Z}$ is a matrix whose rows are unobserved latent factors ${\bf z}_i$
- ${\bf A} \in \mathbb{R}^{p \times q}$ is a matrix whose columns form linear combinations of the factors, ${\bf z}$. These are the factor loadings
- ${\bf E}$ is a normal matrix with 0 mean and Identity covariance
- ${\bf \psi} =\mathrm{diag}(\psi_1, ... ,\psi_p)$. This ensures that all dependencies among the rows of ${\bf Y}$ are dependent on ${\bf Z}$.
Note that $E[{\bf Y}] = {\bf \mathbb{1}\mu}^\top + {\bf ZA}^\top$ and $Var({\bf Y}) = \psi \otimes I$. This model is only useful if $q < p$, otherwise ${\bf ZA}^\top$ is full rank and this model is the same as an unrestricted and arbitrary model, $E[{\bf Y}] = {\bf M}$ for any ${\bf M}$.
