Skip to main content

Factor analysis is a dimensionality reduction latent variable technique which replaces inter-correlating variables with a smaller number of continuous latent variables called factors. The factors are believed to be responsible for the inter-correlations. [For confirmatory factor analysis, please use the tag 'confirmatory-factor'. Also, the term "factor" of factor analysis should not be confused with "factor" as categorical predictor of a regression/ANOVA.]

A latent factor (matrix) model has the following form:

$${\bf Y} = {\bf \mathbb{1}\mu}^\top + {\bf ZA}^\top + {\bf E\psi}^{1/2}$$

Where:

  • ${\bf Y} \in \mathbb{R}^{n \times p}$ whose rows are random vectors in $\mathbb{R}^p$
  • ${\bf \mu} \in \mathbb{R}^p$ is a column vector of means of a generic row of ${\bf Y}$
  • ${\bf Z}$ is a matrix whose rows are unobserved latent factors ${\bf z}_i$
  • ${\bf A} \in \mathbb{R}^{p \times q}$ is a matrix whose columns form linear combinations of the factors, ${\bf z}$. These are the factor loadings
  • ${\bf E}$ is a normal matrix with 0 mean and Identity covariance
  • ${\bf \psi} =\mathrm{diag}(\psi_1, ... ,\psi_p)$. This ensures that all dependencies among the rows of ${\bf Y}$ are dependent on ${\bf Z}$.

Note that $E[{\bf Y}] = {\bf \mathbb{1}\mu}^\top + {\bf ZA}^\top$ and $Var({\bf Y}) = \psi \otimes I$. This model is only useful if $q < p$, otherwise ${\bf ZA}^\top$ is full rank and this model is the same as an unrestricted and arbitrary model, $E[{\bf Y}] = {\bf M}$ for any ${\bf M}$.