PCA and LDA segregate the classes perfectly in 3 dimensions

Question

I'm applying PCA and LDA on my gene expression microarray data to reduce dimensionality. It has ~20,500 features. I tried to visualize the first 3 components from both the methods in a 3d scatter plot and both the methods yielded well separated clusters for the 5 classes of cancer. My goal is to make a classification model, however the 3 components of PCA have explained variance equal to only 0.29 in total. My questions are:

1. Do the rest of the components merely represent noise or redundancy in data, when 3 components sufficiently segregate it? And so can I use the first 3 PCs or LDs only as my input features for the classification model?

2. Should I consider the possibility that classes can be segregated in higher dimensions as well? Is it possible to tell? And if so, in such a case, what would be preferable: lower dimensional segregation with less information or higher dimensional segregation with possible redundancy?

Answer 1

I'm answering my own question, after some research.

Do the rest of the components merely represent noise or redundancy in data, when 3 components sufficiently segregate it?

Not necessarily. It is hard to tell just by looking at plots if the classes are sufficiently segregated. It should be able to generalize well to unseen data, and so you must have a metric to judge the quality of segregation, for example, f1 score.

And so can I use the first 3 PCs or LDs only as my input features for the classification model?

PCA is an unsupervised learning algorithm, and while it does capture the variance in its principal components, it cannot capture the relationship of variables with the response. So choosing only the PCs which explain maximum variance is not advisable when doing regression or classification because you might be throwing away useful information in less variance PCs.

Should I consider the possibility that classes can be segregated in higher dimensions as well? Is it possible to tell?

Yes, like I said above, you must have an objective metric to decide on how well the classes are segregated.

And if so, in such a case, what would be preferable: lower dimensional segregation with less information or higher dimensional segregation with possible redundancy

The number of principal components you choose must be decided by cross validation in case of regression or classification, so you should consider both the options and choose one which gives the best results.