Factor analysis is a dimensionality reduction latent variable technique which replaces inter-correlating variables by a smaller number of continuous latent variables called factors. The factors are believed to be responsible for the inter-correlations. [For confirmatory factor analysis, please use tag 'confirmatory-factor'. Also, term "factor" of factor analysis should not be confused with "factor" as categorical predictor of a regression/ANOVA.]

A latent factor (matrix) model has the following form:

$${\bf Y} = {\bf \mathbb{1}\mu}^T + {\bf ZA}^T + {\bf E\psi}^{1/2}$$


  • $Y \in \mathbb{R}^{n \times p}$ whose rows are random vectors in $\mathbb{R}^p$
  • ${\bf \mu} \in \mathbb{R}^p$ is a vector of column means of $Y$
  • ${\bf Z}$ is a matrix whose rows are unobserved latent factors ${\bf z_i}$
  • ${\bf A} \in \mathbb{R}^{p \times q}$ is a matrix whose columns form linear combinations of the factors, ${\bf z}$. These are the factor loadings
  • ${\bf E}$ is a normal matrix with 0 mean and Identity covariance
  • ${\bf \psi} =diag(\psi_1, ... \psi_p)$. This ensures that all dependency among the rows of ${\bf Y}$ are dependent on ${\bf Z}$.

Note that $E[{\bf Y}] = {\bf \mathbb{1}\mu}^T + {\bf ZA}^T$ and $Var({\bf Y}) = \psi \otimes I$. This model is only useful if $q < p$, otherwise ${\bf ZA}^T$ is full rank and this model is the same as an unrestricted and arbitrary model, $E[{\bf Y}] = {\bf M}$ for any ${\bf M}$.