I am trying to derive an expectation-maximization algorithm for factor analysis.

Let us consider the following model: $$ p(x_i|z_i) = \mathcal N (x_i|Wz_i +\mu, \Psi) $$

$$ p(z_i) = \mathcal N (z_i|0, I) $$ where $W \in \mathbb{R}^{d\times k}$

To derive the EM algorithm we should consider the complete data likelihood, treating the factors $z_i$ as hidden variables. The joint is easily derived.

$$ p(x_i,z_i)=(2\pi)^{-\frac{d+k}{2}} |\Psi|^{-\frac{1}{2}}\exp \left (-\frac{1}{2}(x_i-Wz_i-\mu)^T\Psi^{-1}(x_i-Wz_i-\mu) -\frac{1}{2}z_i^Tz_i\right ) $$ Taking the logarithm of the complete likelihood leads to the following expression (dropping terms independent of the optimized parameters): $$ \ell_c(W,\Psi) = {-\frac{n}{2}}\log|\Psi| -\frac{1}{2}\sum_{i=1}^n (x_i-Wz_i-\mu)^T\Psi^{-1}(x_i-Wz_i-\mu) $$ What bothers me is this. In Ghahramani and Hinton 1996, they propose a derivation of EM for factor analysis, but instead of starting by computing the expected value of the complete data log likelihood, they simply use the log-likelihood of the data conditioned on $z_i$.

This leads to the same result, because the prior does not introduce relevant terms into the optimization problem. However, it does not seem natural to present the derivation this way.

What is the reason for the way the algorithm is derived in the paper? Is there a reason to expect in advance the conditional log-likelihood to work in this case, other than just happening to know that the joint won't make any difference? What is the role of the prior in this approach?


I think the prior plays a role in computing the expectation of $z$ and $zz^\top$, and that presenting Q as the conditional likelihood of y given z is merely because it does not have an effect on $\partial \ell_c/\partial W$ or $\partial \ell_c/\partial \Psi$ in this particular case. You can refer to this (p. 13) for the joint likelihood.

What confuses me, however, is the derivative

$$\frac{\partial \langle\ell_c\rangle}{\partial W}= \frac{\partial}{\partial W}\left(-\dfrac N2 \text{tr}\left[\langle S\rangle\Psi^{-1}\right]\right)$$ where

$$\langle S\rangle=\frac 1N\sum_n\left(x_nx_n^T-x_n\langle Z_n^T\rangle W^T-W\langle Z_n\rangle x_n^T+W\langle Z_nZ_n^T\rangle W^T\right)$$ is the sample covariance matrix. (Notice in the above ppt, the term $W\langle Z_n\rangle x_n^T$ has an extra traonpose on $Z_n$, which I believe is not correct).

Then something like a chain rule takes place

$$\frac{\partial \langle\ell_c\rangle}{\partial W}=\frac{\partial \langle\ell_c\rangle}{\partial \langle S\rangle}\cdot \frac{\partial \langle S\rangle}W=\Psi^{-1}\cdot\left( \sum_n x_n\langle Z_n^T\rangle - W\sum_n\langle Z_nZ_n^T\rangle \right) $$

I do not understand how $\frac{\partial \langle S\rangle}W$ is computed, and how this rank-four tensor is multiplied with $\dfrac{\partial \langle\ell_c\rangle}{\partial \langle S\rangle}$.

Have you worked this out? Or do you have any matrix calculus cheatsheet containing rules relevant to the above derivation? I took a look at wikipeida, but it seems no rules can be applied here.

Thank you

  • $\begingroup$ Thanks for the pointers. I'll take a closer look as soon as I have some free time. I haven't gone back to this issue for a while now, but it is still in my backlog. If I get a good understanding of it, I'll try to post my findings. $\endgroup$ – cangrejo Nov 22 '17 at 19:21

TLDR; The reason for why they start with the conditional likelihood rather then the complete likelihood is that the terms that do not involve the parameters of interest can be treated constant when maximizing the evidence lower-bound - adding a constant $C$ to any function does not change its shape and therefore optima.

The authors use Expectation-Maximization (EM) algorithm, which iteratively maximizes a lower-bound to the observed-data log-likelihood also called ELBO (for evidence lower bound), which is where the complete data log-likelihood appears $$ \begin{align} \log p(x_i; \theta) &\geq \mathbb{E}_{p(z_i \mid x_i; \theta^{(t-1)})}\left[ \log \frac{p(x_i \mid z_i; \theta) p(z_i)}{p(z_i \mid x_i; \theta^{(t-1)})} \right]\\ &= \mathbb{E}_{p(z_i \mid x_i; \theta^{(t-1)})}\left[ \log p(x_i \mid z_i; \theta) \right] + \log p(z_i) - \mathbb{E}_{p(z_i \mid x_i; \theta^{(t-1)})}\left[ \log p(z_i \mid x_i; \theta^{(t-1)}) \right] \\ &= Q(\theta; \theta^{(t-1)}), \end{align} $$ where I have denoted $\theta = (\mu, W, \Psi)$, and $\theta^{(t-1)}$ denotes the parameter estimates from the previous iteration. $Q(\theta; \theta^{(t-1)})$ here is a function that you want to maximize wrt $\theta$ at each iteration $t$.

Maximizing $Q(\theta; \theta^{(t-1)})$ wrt $\theta$, and keeping $\theta^{(t-1)}$ fixed, means that the negative term above (also called the conditional entropy) and $\log p(z_i)$ are both constant for all $\theta$. Hence you have $$ \begin{align} \theta^{(t)} &= \operatorname*{arg\,max}_{\theta} Q(\theta; \theta^{(t-1)})\\ &= \operatorname*{arg\,max}_{\theta} \mathbb{E}_{p(z_i \mid x_i; \theta^{(t-1)})}\left[ \log p(x_i \mid z_i; \theta) \right] + C \\ &= \operatorname*{arg\,max}_{\theta} \mathbb{E}_{p(z_i \mid x_i; \theta^{(t-1)})}\left[ \log p(x_i \mid z_i; \theta) \right], \end{align} $$ where I used $C$ to capture the constant terms. So the actual function you end up maximizing requires only the conditional log-likelihood.

To answer your second question "What is the role of the prior in this approach?", in a Factor Analysis model any variation in the prior $p(z)$ can be absorbed into the factor loading matrix $W$, so in practice using the identity covariance is usually sufficient, making computations much easier.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.