I'm reading some notes in multivariate data analysis, in particular factor analysis.

  • A data vector $X_{p\times 1}$, with $E(X) = \mu$

  • A vector $F_{m \times 1} $ of factors,

  • A matrix $L_{p\times m}$ of loadings, and

  • A vector $\varepsilon_{p \times 1}$ of $p$ errors with a diagonal covariance matrix $Var(\varepsilon) = \Psi$

which gives the model

$$ X-\mu = LF + \varepsilon $$

The model has $mp + p$ parameters after considering $L$ and $\Psi$.

The solutions are not unique, and are only determined up to orthogonal rotations $L^* = LT$ where $T$ is an orthogonal matrix. This is used profitably to rotate the factors in a way that provides better interpretation.


Then, the notes say:

after rotating with $T$, there are $\frac{m(m-1)}{2}$ fewer parameters.

I just can't figure out where that comes from. Can someone please explain?

  • $\begingroup$ @amoeba No, you're right, I probably don't want it there. However, I would appreciate either a hint or a full solution. Whichever gets me closer to understanding. $\endgroup$ – RMurphy Oct 17 '16 at 23:58
  • $\begingroup$ I don't have time for a full answer right now, but here is a hint: how many parameters does $T$ have? $\endgroup$ – amoeba Oct 18 '16 at 0:06
  • $\begingroup$ @amoeba Hmm... I thought it was fixed and known, based on what I've read so far. I'll continue reading, and keep that hint in mind. $\endgroup$ – RMurphy Oct 18 '16 at 1:42
  • 1
    $\begingroup$ Hi, I think I figured one thing out. Observe that, in order for a matrix $T$ to be orthogonal, its columns need to sum to one be mutually orthogonal. Hence, that gives a system of equations that the elements of the matrix need to satisfy. I found that $\frac{m(m-1)}{2}$ parameters are freely varying, in the sense that if you wanted to make an orthogonal matrix, you could start by picking $\frac{m(m-1)}{2}$ elements but then the rest would be determined. So I kind of get that. Is that the right direction? $\endgroup$ – RMurphy Oct 18 '16 at 18:10
  • 1
    $\begingroup$ @amoeba I understand that $T$ has $m(m-1)/2$ free parameters and that L is only unique to multiplication by an orthogonal matrix. However, I am unable to understand how these two facts put together reduces the number of free parameters in $L$ by $m(m-1)/2$? $\endgroup$ – honeybadger Jul 31 '19 at 17:32

I found the following references - Rotation in Factor analysis by Darton and this presentation wherein it is stated that because of the rotational indeterminacy, an extra constraint is imposed - primarily for computational convenience.

The extra constraint is usually of the form: $L^{T} \Psi^{-1} L$ is a diagonal matrix. By construction $L^{T} \Psi^{-1} L$ is a symmetric matrix, and setting the non-diagonal elements to zero is equivalent to a reduction in $m(m-1)/2$ degrees of freedom.


This answer by Amoeba explains it in more detail.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.