Classifying by performing PCA for two classes separately I have a dataset with binary labels, and I try to figure out whether the data can be classified and yield the ground-truth labels. I thought to try PCA for the data with each of the labels, and see whether I get a different PCA basis / coefficients. This is under the assumption that if the data is not differentiable, I would (probably) get the PCA coefficients to have similar shapes. 
The question is, is this method valid? If I do get different PCA coefficients for the two groups, does this mean they have different statistical properties? 
 A: If two classes come from the same distribution, then yes, separate PCAs should yield similar results. But the opposite is not true! PCA analyzes covariance structure of the data, which means that it ignores the mean. So the two classes can be 100% linearly separated (leading to 100% classification accuracy with a linear method), but still have identical within-class covariances, e.g.:

Therefore your approach does not seem to make a lot of sense.
A: Adding to amoeba's answer, I will give a sketch of a principled way to perform classification using probabilistic PCA. pPCA is a model of the form
$$
p(x) = \mathcal{N}(\mu, C)
$$
where $\mu = \mathbb{E}[x]$ and $C = WWT + \sigma^2 I$. Finding the parameters (i.e, $\mu, W, \sigma^2$) can then be done by maximum likelihood. If $\sigma^2 \rightarrow 0$, the standard PCA model is recovered. Note that this model includes the mean, though.
Now, a classification rule can be obtained by making use Bayes formula. We estimate parameters for each class $i$ separately and can get:
$$p(c_i|x) = {p(x|c_i)p(c_i) \over p(x)},$$
where $p(c)$ are the class priors and $p(x|c)$ represents the class specific PCA. This is an example of a generative model for classification.
Some intuition is as follows. Assume both classes are equally likely (e.g. $p(c_i) \propto 1$). 
If $C_i = I$, we will just assign each point to the class with the closest mean. If $C_i = C_j \forall i, j$, the corresponding Mahalanobis distance will be used. In the general case, we will calculate the class specific Mahalanobis distance from the class specific mean and pick the class for which this value is lowest. 
