4
$\begingroup$

I am trying to derive an expectation-maximization algorithm for factor analysis.

Let us consider the following model: $$ p(x_i|z_i) = \mathcal N (x_i|Wz_i +\mu, \Psi) $$

$$ p(z_i) = \mathcal N (z_i|0, I) $$ where $W \in \mathbb{R}^{d\times k}$

To derive the EM algorithm we should consider the complete data likelihood, treating the factors $z_i$ as hidden variables. The joint is easily derived.

$$ p(x_i,z_i)=(2\pi)^{-\frac{d+k}{2}} |\Psi|^{-\frac{1}{2}}\exp \left (-\frac{1}{2}(x_i-Wz_i-\mu)^T\Psi^{-1}(x_i-Wz_i-\mu) -\frac{1}{2}z_i^Tz_i\right ) $$ Taking the logarithm of the complete likelihood leads to the following expression (dropping terms independent of the optimized parameters): $$ \ell_c(W,\Psi) = |\Psi|^{-\frac{n}{2}} -\frac{1}{2}\sum_i (x_i-Wz_i-\mu)^T\Psi^{-1}(x_i-Wz_i-\mu) $$ What bothers me is this. In Ghahramani and Hinton 1996, they propose a derivation of EM for factor analysis, but instead of starting by computing the expected value of the complete data log likelihood, they simply use the log-likelihood of the data conditioned on $z_i$.

This leads to the same result, because the prior does not introduce relevant terms into the optimization problem. However, it does not seem natural to present the derivation this way.

What is the reason for the way the algorithm is derived in the paper? Is there a reason to expect in advance the conditional log-likelihood to work in this case, other than just happening to know that the joint won't make any difference? What is the role of the prior in this approach?

$\endgroup$
1
$\begingroup$

I think the prior plays a role in computing the expectation of $z$ and $zz^\top$, and that presenting Q as the conditional likelihood of y given z is merely because it does not have an effect on $\partial \ell_c/\partial W$ or $\partial \ell_c/\partial \Psi$ in this particular case. You can refer to this (p. 13) for the joint likelihood.

What confuses me, however, is the derivative

$$\frac{\partial \langle\ell_c\rangle}{\partial W}= \frac{\partial}{\partial W}\left(-\dfrac N2 \text{tr}\left[\langle S\rangle\Psi^{-1}\right]\right)$$ where

$$\langle S\rangle=\frac 1N\sum_n\left(x_nx_n^T-x_n\langle Z_n^T\rangle W^T-W\langle Z_n\rangle x_n^T+W\langle Z_nZ_n^T\rangle W^T\right)$$ is the sample covariance matrix. (Notice in the above ppt, the term $W\langle Z_n\rangle x_n^T$ has an extra traonpose on $Z_n$, which I believe is not correct).

Then something like a chain rule takes place

$$\frac{\partial \langle\ell_c\rangle}{\partial W}=\frac{\partial \langle\ell_c\rangle}{\partial \langle S\rangle}\cdot \frac{\partial \langle S\rangle}W=\Psi^{-1}\cdot\left( \sum_n x_n\langle Z_n^T\rangle - W\sum_n\langle Z_nZ_n^T\rangle \right) $$

I do not understand how $\frac{\partial \langle S\rangle}W$ is computed, and how this rank-four tensor is multiplied with $\dfrac{\partial \langle\ell_c\rangle}{\partial \langle S\rangle}$.

Have you worked this out? Or do you have any matrix calculus cheatsheet containing rules relevant to the above derivation? I took a look at wikipeida, but it seems no rules can be applied here.

Thank you

$\endgroup$
  • $\begingroup$ Thanks for the pointers. I'll take a closer look as soon as I have some free time. I haven't gone back to this issue for a while now, but it is still in my backlog. If I get a good understanding of it, I'll try to post my findings. $\endgroup$ – broncoAbierto Nov 22 '17 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.