2
$\begingroup$

Suppose we have $n$, $p$-dimensional, samples $\overrightarrow{X_i} \sim \mathcal{N}(\mu, \Psi+\mathbf{w^Tw})$. $\Psi$ is a diagonal matrix of specific variances, while $\mathbf{w^Tw}$ composes the remainder covariance structure in terms of factor loadings, $\mathbf{w}$. Under this model, $\mathbf X=\mu+\mathbf{Fw}+\epsilon$, where $\mathbf F$ denotes the (orthogonal) scores pertaining to $q$ factors $(q\lt p)$.

It's log-likelihood then is:

$$\mathcal{LL}=-{np\over2}\log(2\pi)-{n\over2}\log|\Psi+\mathbf{w^Tw}|-{n\over2}\text{tr}\left(\left(\Psi+\mathbf{w^Tw}\right)^{-1}\mathbf V\right)$$

Where

$$\mathbf V={(\mathbf X-\mu)^T(\mathbf X-\mu)\over n}$$

In these notes (under equation 27) it is stated that:

  • A: starting from an estimate of $\Psi$ (an initial guess is easily obtained from multiple linear regression $R^2$s), the optimal $\Psi^{1/2}\mathbf w^T$ is given by the leading $q$ ($q\lt p$) eigen-vectors of $\Psi^{1/2}\mathbf V\Psi^{1/2}$
  • B: likewise, starting from an estimate of $\mathbf w$, the optimal choice for $\Psi$ is $\mathbf {V - w^Tw}$. This can be derived from the definition of $\mathbf V$

Having both of these definitions, a two-stage MLE can be performed iteratively, until convergence.

My question is, how was A actually derived? I couldn't make the connection between $\Psi^{1/2}\mathbf w^T$ and $\Psi^{1/2}\mathbf V\Psi^{1/2}$.


By the way, here are the partial derivatives of the log-likelihood (unless I made some mistake):

\begin{align} &\frac{\partial\mathcal{LL}}{\partial\mathcal{\mathbf \Psi}}= (\Psi+\mathbf{w^Tw})^{-1}-(\Psi+\mathbf{w^Tw})^{-1}\mathbf V (\Psi+\mathbf{w^Tw})^{-1} \\ &\frac{\partial\mathcal{LL}}{\partial\mathcal{\mathbf w}}=2\mathbf w \frac{\partial\mathcal{LL}}{\partial\mathcal{\mathbf \Psi}} \end{align}

$\endgroup$
  • $\begingroup$ See that I followed the reference given in the notes and, while we reach a workable solution in the answer, it's given in terms of other elements! I'll accept an answer that arrives at the same eigen-decomposition described in the notes :) $\endgroup$ – Firebug Aug 23 '19 at 23:17
3
$\begingroup$

The proof below is provided by Bartholomew, Knott and Moustaki (2011)


Starting from the derivatives of the log-likelihood:

\begin{align} &\frac{\partial\mathcal{LL}}{\partial\mathcal{\mathbf \Psi}}= (\Psi+\mathbf{w^Tw})^{-1}-(\Psi+\mathbf{w^Tw})^{-1}\mathbf V (\Psi+\mathbf{w^Tw})^{-1} \\ &\frac{\partial\mathcal{LL}}{\partial{\mathbf w}}=2\mathbf w \frac{\partial\mathcal{LL}}{\partial\mathcal{\mathbf \Psi}} \end{align}

The stationarity condition for $\mathbf w$ is

$${1\over2}\frac{\partial\mathcal{LL}}{\partial{\mathbf w}}= \mathbf w \left( (\Psi+\mathbf{w^Tw})^{-1}-(\Psi+\mathbf{w^Tw})^{-1}\mathbf V (\Psi+\mathbf{w^Tw})^{-1} \right)=\mathbf 0$$

If we right-multiply by $(\Psi+\mathbf{w^Tw})\mathbf V^{-1}(\Psi+\mathbf{w^Tw})\mathbf V^{-1/2}$:

$$\mathbf w (\mathbf V^{-1}(\Psi+\mathbf{w^Tw})\mathbf V^{-1/2}+\mathbf V^{-1/2}) = \mathbf w \mathbf V^{-1/2}(\mathbf V^{-1/2}(\Psi+\mathbf{w^Tw})\mathbf V^{-1/2}-\mathbb I) =\mathbf 0$$

Rearranging terms:

$$ \mathbf w \mathbf V^{-1/2}(\mathbf V^{-1/2}\Psi\mathbf V^{-1/2}) = (\mathbb I -\mathbf{w}\mathbf V^{-1}\mathbf{w^T})\mathbf w \mathbf V^{-1/2}$$

If $\mathbf{w}\mathbf V^{-1}\mathbf{w^T}$ is made diagonal, if becomes clear that $\mathbf w \mathbf V^{-1/2}$ has columns that are eigenvectors of $\mathbf V^{-1/2}\Psi\mathbf V^{-1/2}$, since the last equation is the eigen-decomposition of it.


Notice that I followed the reference given in the notes and, while we reach a workable solution in the answer, it's given in terms of other elements!


Bartholomew, D. , Knott, M. and Moustaki, I. (2011). The Normal Linear Factor Model. In Latent Variable Models and Factor Analysis (eds W. A. Shewhart, S. S. Wilks, D. Bartholomew, M. Knott and I. Moustaki). doi:10.1002/9781119970583.ch3

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.