# Classifying by performing PCA for two classes separately

I have a dataset with binary labels, and I try to figure out whether the data can be classified and yield the ground-truth labels. I thought to try PCA for the data with each of the labels, and see whether I get a different PCA basis / coefficients. This is under the assumption that if the data is not differentiable, I would (probably) get the PCA coefficients to have similar shapes.

The question is, is this method valid? If I do get different PCA coefficients for the two groups, does this mean they have different statistical properties?

• What statistical properties? And what made you think that PCA (which is just a special case of rotation) will help? – ttnphns Oct 7 '14 at 6:45
• The PCA coeffs are the eigenvectors of the covariance matrix, so I would expect that if the data cannot be classified (came from the same distribution), then their PCA transform will yield the same coefficients, and otherwise - I would get different PCA coeffs. – yoki Oct 7 '14 at 8:03

If two classes come from the same distribution, then yes, separate PCAs should yield similar results. But the opposite is not true! PCA analyzes covariance structure of the data, which means that it ignores the mean. So the two classes can be 100% linearly separated (leading to 100% classification accuracy with a linear method), but still have identical within-class covariances, e.g.:

Therefore your approach does not seem to make a lot of sense.

• Yes, it ignores the means. But if even after this, I get different PCAs, it means that the covariance structure of the two classes is indeed different, right? Thanks for the response. – yoki Oct 7 '14 at 11:17
• Well, yes, but this is unlikely to be a very useful information. Difference in means is usually much more important for classification purposes than difference in covariance. Imagine blue and red clusters on my figure having the same mean, but one stretched horizontally and another -- vertically. No linear method for classification will exceed chance performance, and even nonlinear methods will not be very successful (even though the two PCAs will be as different as it gets). – amoeba says Reinstate Monica Oct 7 '14 at 11:22
• I have data which appear to be very similar - there is no difference in the means when I look at combinations of features. However, when I do PCA for each separately, I do see different "clusters" in the PCA coeff space. The question is how to go on from here - how to classify using this information. – yoki Oct 7 '14 at 11:41
• I am not sure what you mean by "clusters in the PCA coeff space", but my point in the previous comment was that you can easily have leading PCA axes wildly different between your classes, and at the same time very poor (or even chance level) classification. In any case, if the means are the same, you will need to use nonlinear methods. There are plenty. – amoeba says Reinstate Monica Oct 7 '14 at 11:49
• An important contrast/distinction could be made here to Fisher's linear and quadratic discriminants, which attempt to rotate the mean vectors in a way that maximizes class separation. – Andrew M Dec 15 '14 at 23:38

Adding to amoeba's answer, I will give a sketch of a principled way to perform classification using probabilistic PCA. pPCA is a model of the form $$p(x) = \mathcal{N}(\mu, C)$$ where $\mu = \mathbb{E}[x]$ and $C = WWT + \sigma^2 I$. Finding the parameters (i.e, $\mu, W, \sigma^2$) can then be done by maximum likelihood. If $\sigma^2 \rightarrow 0$, the standard PCA model is recovered. Note that this model includes the mean, though.

Now, a classification rule can be obtained by making use Bayes formula. We estimate parameters for each class $i$ separately and can get: $$p(c_i|x) = {p(x|c_i)p(c_i) \over p(x)},$$ where $p(c)$ are the class priors and $p(x|c)$ represents the class specific PCA. This is an example of a generative model for classification.

Some intuition is as follows. Assume both classes are equally likely (e.g. $p(c_i) \propto 1$).

If $C_i = I$, we will just assign each point to the class with the closest mean. If $C_i = C_j \forall i, j$, the corresponding Mahalanobis distance will be used. In the general case, we will calculate the class specific Mahalanobis distance from the class specific mean and pick the class for which this value is lowest.

• What's the advantage of using PPCA here at all, as opposed to using class means $\boldsymbol \mu_i$ and within-class covariances $\mathbf C_i$ directly? – amoeba says Reinstate Monica Oct 7 '14 at 11:29
• Can you please cite a paper that uses this method? – yoki Oct 7 '14 at 11:44
• @amoeba, PPCA is a generalisation of PCA. There is no general advantage, apart from making it probabilistic and thus enabling the generative model approach. Using the $\mu_i$ and $C_i$ directly is fine, but adding $\sigma_i^2$ might help represent your data better. – bayerj Oct 7 '14 at 12:28
• @ida I don't know of any paper, but Bishop's "Pattern recognition and machine learning" explains both pPCA and generative models for classification. This is just a combination of the two, which is fairly standard. – bayerj Oct 7 '14 at 12:29
• Ah, I get it. You are right, there is no strict need for pPCA here. I was just assuming that the OP has a good reason why to do it. (E.g. computational efficiency or reducing the risk of overfitting.) – bayerj Oct 7 '14 at 20:03