I am trying to investigate using principal component analysis whether it is possible to guess with good confidence from which population ("Aurignacian" or "Gravettian") a new datapoint came from. A datapoint is described by 28 variables, most of which are relative frequencies of archaeological artefacts. The remaining variables are computed as ratios of other variables.
Using all variables, the populations partially segregate (subplot (a)), but there is still some overlap in their distribution (90% t-distribution prediction ellipses, though I am not sure I can assume normal distribution of populations). I thought it was therefore not possible to predict with good confidence the origin of a new datapoint:
Removing one variable (r-BEs), the overlap becomes much more important, (subplots (d), (e), and (f)), as the populations do not segregate in any paired PCA plots: 1-2, 3-4, ..., 25-26, and 1-27. I took this to mean r-BEs was essential for separating the two populations, because I thought that taken together, these PCA plots represent 100% of the "information" (variance) in the dataset.
I was hence extremely surprised to notice that the populations actually did segregate almost completely if I dropped all but a handful of variables:
Why is this pattern not visible when I perform a PCA on all variables? With 28 variables, there are 268,435,427 ways of dropping a bunch of them. How may one find those that will maximise population segregation and best allow guessing the population of origin of new datapoints? More generally, is there a systematic way of finding "hidden" patterns like these?
EDIT: Per amoeba's request, here are the plots when the PCs are scaled. The pattern is clearer. (I realise I'm being naughty by continuing to knock out variables, but the pattern this time resists to the knock-out of r-BEs, implying the "hidden" pattern is picked up by the scaling):