In PCA, if I have a latent $\vec{y}$ with loading matrix $\Lambda$, then the PCA models using:

(1) $P(\vec{y}) \sim N(\vec{0}, I)$, $P(\vec{x}|\vec{y}) \sim N(\Lambda \vec{y}, \psi I)$

(2) $P(\vec{y}) \sim N(\vec{0}, \Gamma), P(\vec{x}|\vec{y}) \sim N(\Lambda \vec{y}, \psi I)$ with $\Gamma_{ij}=0$ for $i \neq j$ and $\Lambda^T \Lambda = I$

lead to exactly the same marginal distributions.

So far, I have attempted to compute the marginal of (1) by integrating out over $\vec{y}$ as follows (I am assuming the latent variable $\vec{y}$ has dimension $n$ and the normal variable $\vec{x}$ has dimension $D$):

$P(\vec{x}) = \int_{\mathbb{R}^n} P(\vec{x}| \vec{y}) P(\vec{y}) d \vec{y}= \int_{\mathbb{R}^n} \frac{1}{\sqrt{(2 \pi)^n | I|}}exp(-\frac{1}{2} \vec{y}^T \vec{y}) P(\vec{x}|\vec{y}) d \vec{y} = \int_{\mathbb{R}^n} \frac{1}{(2 \pi)^{\frac{n+D}{2}}\psi^{\frac{1}{2}}|I|}exp(-\frac{1}{2} \vec{y}^T \vec{y}) exp \left(-\frac{1}{2 \psi} [\vec{x}^T \vec{x} - 2 \vec{x}^T \Lambda \vec{y} + (\Lambda \vec{y})^T (\Lambda \vec{y})] \right) d \vec{y}$

I am not sure if I am on the right lines? If I am, what should I do next? Can I simplify this marginal for (1) any more or should I go ahead and try and bring the marginal of (2) to this form?



lead to exactly the same marginal distributions, I don't think so.

Since both the marginal and conditional are normal, the marginal of $x$ is also normal and you can easily find the moments using the iterated expectations:

$E(x) = E(E(x|y)),$

$Var(x) = E(var(x|y)) + var(E(x|y)).$

If you want to continue from the integral you sated, to solve it: first, get rid of all quantities that contribute nothing, like the determinant of the identity matrix, which equals one. Next, remove from the integral all quantities that do not involve $y$. You will be left with an exponential, which you can make look like the exponential of a multivariate normal with certain mean and covariance matrix. Solving the latter is done using the general property that distribution integrate to one.

I hope this helps.

  • $\begingroup$ Hi @papgeo thanks for your reply. I believe the exact wording of the question was "show that this alternative model (2) is equivalent to the standard one (1) in the sense that it can model exactly the same set of possible marginal distributions". Does this change anything? I'm not really sure how to address the "distributions it can model" part so I just assumed it meant they had the same distribution.... $\endgroup$ – user11128 Feb 18 at 11:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.