The conventional model for probabilistic principal component analysis has a standard normal latent $\vec{y}$ and a loading matrix $\Lambda$:

$P(\vec{y}) \sim N(\vec{0}, I)$, $P(\vec{x}|\vec{y}) \sim N(\Lambda \vec{y}, \psi I)$

An alternative would be to draw $\vec{y}$ from a normal with diagonal covariance (say $\Sigma$, and then restrict $\Lambda$ to have orthonormal columns:

$P(\vec{y}) \sim N(\vec{0}, \Gamma), P(\vec{x}|\vec{y}) \sim N(\Lambda \vec{y}, \psi I)$ with $\Gamma_{ij}=0$ for $i \neq j$ and $\Lambda^T \Lambda = I$

where $\Gamma_{ij} =\Gamma_{ji} =0$ and $\Lambda^T \Lambda=0$.

Question: Show that this alternative model is equivalent to the standard one in the sense that it can model exactly the same set of possible marginal distributions.

I really have no idea what I am being asked to do here. I know that there is some equivalence of PCA models under rotation and scaling - should I be working in that direction, or am I supposed to integrate out the latent variable and show I get the same distribution? Or something else entirely? Help please.

lead to exactly the same marginal distributions.


1 Answer 1


Here are some hints. I will rename $\Lambda$ in the first model to $A$ to avoid confusion with the one that is to have orthonormal columns.

As you seem to have deduced, the distributions we get for $x$ are $N(0,\psi I + \Lambda \Gamma \Lambda^T)$ in one model and $N(0,\psi I + AA^T)$ in the other model.

Therefore, it suffices to show that matrices of the form $\Lambda \Gamma \Lambda^T$ (with $\Lambda$ having orthonormal columns and $\Gamma$ diagonal) and $AA^T$ are equivalent (either can be written in the other's form).

To help show that $\Lambda \Gamma \Lambda^T$ can be written as $AA^T$, utilise the square root of $\Gamma$. To help show that $AA^T$ can be written as $\Lambda \Gamma \Lambda^T$, utilise the spectral theorem and drop eigenvectors corresponding to $0$ eigenvalues (and drop these eigenvalues from the diagonal matrix from the spectral theorem).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.