Probabilistic PCA - Gaussian: In their Probabilistic PCA model, Tipping and Bishop assume the following model

$$ \boldsymbol{x} \sim \mathcal{N}(0, \mathbf{I})\\ \mathbf{t} | \boldsymbol{x} \sim \mathcal{N}(\boldsymbol{Wx}+\boldsymbol{\mu}, \sigma^2\mathbf{I}) $$

or equivalently $$ \mathbf{t} \sim \mathcal{N}(\boldsymbol{\mu}, \sigma^2\mathbf{I} + \mathbf{WW^T}) $$

and noticed that "Under this model, the Maximum Likelihood Estimator $W_{ML}$ is that matrix whose columns are the scaled and rotated principal eigenvectors of the sample covariance matrix.".

In other words, the matrix $\mathbf{W}$ is an orthogonal basis.

Probabilistic PCA - Bernoulli: Then Tipping replaced the conditional likelihood of the first equations by a Bernoulli: $$ \boldsymbol{x} \sim \mathcal{N}(0, \mathbf{I})\\ \mathbf{t} | \boldsymbol{x} \sim \mathcal{Bernoulli}(\boldsymbol{Wx}+\mathbf{b}) $$ where b is a bias term.

Probabilistic PCA - exponential family: Later, Collins, generalized it to the exponential family. It looks like it assumes

$$ \mathbf{t} \sim f(\boldsymbol{Av}) $$

where $\mathbf{Av}$ is the natural parameter of the distribution.

Do the Bernoulli and the exponential family cases also assume orthogonal basis? Or they just lose the connection with the PCA framework and become just Matrix Factorization models?

  • $\begingroup$ +1. Note that if you train [Gaussian] PPCA with EM, then you will not have orthogonal columns in W. See the last sentence in section 2. $\endgroup$ – amoeba Oct 22 '18 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.