I am currently reading papers about probabilistic PCA and I am wondering why is Gaussian prior (and not some other prior) chosen for the latent variables? Is it just because it's simple or is there another reason?



Probabilistic PCA

Probabilistic PCA is a Gaussian latent variable model of the following form. Observations $\mathbf x \in \mathbb R^D$ consist of $D$ variables, latent variables $\mathbf z \in \mathbb R^M$ are assumed to consist of $M<D$ variables; the prior over latent variables is a zero-mean unit-covariance Gaussian: $$\mathbf z \sim \mathcal N(\mathbf 0, \mathbf I),$$ and the conditional distribution of observed variables given the latent variables is $$\mathbf x | \mathbf z \sim \mathcal N(\mathbf W\mathbf z+\boldsymbol \mu, \sigma^2 \mathbf I).$$ It turns out that the maximum likelihood solution to this model is given by the first $M$ PCA components of the data: columns of $\mathbf W_\text{ML}$ are proportional to the top eigenvectors of the covariance matrix (principal axes). See Tipping & Bishop for details.

Why using Gaussian prior?

  1. For any other prior (or at least for most other priors) the maximum likelihood solution will not correspond to the standard PCA solution, so there would be no reason to call this latent variable model "probabilistic PCA". Gaussian $\mathcal N(\mathbf 0, \mathbf I)$ prior is the one that gives rise to PCA.

  2. Most other priors would make the problem much more complicated or even analytically intractable. Having Gaussian prior and Gaussian conditional distribution leads to Gaussian marginal distribution $p(\mathbf x)$, and it is easy to see that its covariance matrix will be given by $\mathbf W^\top \mathbf W + \sigma^2\mathbf I$. Non-Gaussian distributions are much harder to work with.

  3. Having Gaussian marginal distribution $p(\mathbf x)$ is also attractive because the task of standard PCA is to model the covariance matrix (i.e. the second moment); PCA is not interested in higher moments of the data distribution. The Gaussian distribution is fully described by the first two moments: mean and covariance. We don't want to use more complicated/flexible distributions, because PCA is not dealing with these aspects of the data.

  4. The Gaussian prior has unit covariance matrix because the idea is to have uncorrelated latent variables that give rise to the observed covariances only via loadings $\mathbf W$.

  • $\begingroup$ Thank you ! It is really clear ! For the first point i agree but its seems to be an answer to the question 'Why this model is called PPCA ?' The point 2 to 4 are exactly what i was expecting for, i should have turned the question into 'What are the benefits of taking a gaussian prior ?' $\endgroup$
    – Irminsul
    Jan 13 '16 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.