PCA for observations subsampling before mRMR feature selection affects downstream Random Forest classification

Question

I have a dataset composed by a matrix of $m$ observations and $n$ predictors (features), where each observation is accompanied by a classification label.

My objective is to train a Random Forest classifier on matrix and classification label array.

The distribution of classification labels is strongly unbalanced, therefore I decided to proceed with downsizing all classes to the size of the minority class. Keeping this in mind, I had two ideas:

• random subsampling of $k$ observations from each class (drawing without replacement)
• Principal Component Analysis (PCA) on matrix samples to get $k$ meta-observations (principal components) from each class - my idea was to get the $k$ most representatives observations which could describe each class

Given that the size of the minority class is $204$, and I have two classes, I set the final number of observations to retain to $408$.

From the side of feature selection, I decided to use the mRMR algorithm [1-2] which seems to be suitable to get the minimum number of features which are most relevant to predict a classification variable. It makes use of mutual information, minimizing the mutual information between each pair of features and maximizing the mutual information between each feature and the classification variable.

Regarding the number of features to retain, I read from [3] that a robust rule of thumb is to retain no more than $m/5$ features, where $m$ is the number of observations. Therefore, given that I want to have $408$ observations, I shall not select more than $81$ features.

Keeping this in mind, I did two experiments:

1. In the first one I used mRMR to get $81$ features, then random subsampling of $408$ observations, then Random Forest classification
2. In the second one I first reduced the number of observations to $408$ with PCA (I could not do it after mRMR $81$-features selection because it would lead to only $81$ principal components), then I used mRMR to get $81$ features, finally Random Forest classification

At this point, my expectations were to see an increase of performance int the second experiment w.r.t. the first experiments. Instead, in the first experiment I get a macro-averaged ROC/AUC score of $92$%, while in the second I reach only $70$%.

Why the second experiment does not work as I expected?

1. Peng, Hanchuan, Fuhui Long, and Chris Ding. "Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy." IEEE Transactions on pattern analysis and machine intelligence 27.8 (2005): 1226-1238.
2. Ding, Chris, and Hanchuan Peng. "Minimum redundancy feature selection from microarray gene expression data." Journal of bioinformatics and computational biology 3.02 (2005): 185-205.
3. Johnstone, Iain M., and D. Michael Titterington. "Statistical challenges of high-dimensional data." Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 367.1906 (2009): 4237-4253.

Answer 1

PCA cannot perform "subsampling" or selecting "most representative observations". It is not selecting anything, it is constructing linear combinations with maximum variance. In your case, PCA will be constructing linear combinations of observations, which has unclear meaning and hardly predictable consequences for your algorithm.

You should be better off with random subsampling.