PCA as Pre-Processing before Clustering through GMM

Question

Suppose I start with a data matrix $X \in \mathbb{R}^{N \times D}$, where each row $x_i$ is $D$-dimensional sample. I would like to cluster this data through a Gaussian Mixture Model (GMM).

If I pre-process the data using PCA, then I can attain a new data matrix $X' \in \mathbb{R}^{N\times D}$, where each variable (column) are orthogonal to each other. I think this implies that the covariance matrix of this transformed data is diagonal.

My questions are

• Would it make sense to assume that the components in my GMM have diagonal covariance matrices?
• Is there any relationship between this process and Probabilistic PCA (PPCA)?
• Does this process make sense / when would I want to use PCA before clustering in GMM?

Answer 1

1. Would it make sense to assume that the components in my GMM have diagonal covariance matrices?

No. Define a $K=2$ mixture of bi-variate Gaussian distributions with covariance matrices that are not diagonal:

$$ f(X)=\sum_{k=1}^K\pi_k\mathcal{N}_k(\mathbf{0},\boldsymbol{\Sigma}_k) $$ where: $\pi_1=\pi_2=0.5$ $$ \boldsymbol{\Sigma}_1= \begin{pmatrix} 2,1\\ 1,2 \end{pmatrix} $$ and $$ \boldsymbol{\Sigma}_2= \begin{pmatrix} 2,-1\\ -1,2 \end{pmatrix} $$ My intuition tells me that the overall covariance matrix $Var(X)$ would be diagonal.

2. Is there any relationship between this process and Probabilistic PCA (PPCA)?

Sort of. PPCA is a PCA equivalent using MLE instead of SVD for solution. In this way, PPCA is related to your process.

3. Does this process make sense / when would I want to use PCA before clustering in GMM?

It makes sense since conventional GMM can not handle high dimensional data. But we have better ways of doing it. You can check Mixture of Probabilistic PCA:

http://www.miketipping.com/papers/met-mppca.pdf