1
$\begingroup$

I have a dataset composed by a matrix of $m$ observations and $n$ predictors (features), where each observation is accompanied by a classification label.

My objective is to train a Random Forest classifier on matrix and classification label array.

The distribution of classification labels is strongly unbalanced, therefore I decided to proceed with downsizing all classes to the size of the minority class. Keeping this in mind, I had two ideas:

  • random subsampling of $k$ observations from each class (drawing without replacement)
  • Principal Component Analysis (PCA) on matrix samples to get $k$ meta-observations (principal components) from each class - my idea was to get the $k$ most representatives observations which could describe each class

Given that the size of the minority class is $204$, and I have two classes, I set the final number of observations to retain to $408$.

From the side of feature selection, I decided to use the mRMR algorithm [1-2] which seems to be suitable to get the minimum number of features which are most relevant to predict a classification variable. It makes use of mutual information, minimizing the mutual information between each pair of features and maximizing the mutual information between each feature and the classification variable.

Regarding the number of features to retain, I read from [3] that a robust rule of thumb is to retain no more than $m/5$ features, where $m$ is the number of observations. Therefore, given that I want to have $408$ observations, I shall not select more than $81$ features.

Keeping this in mind, I did two experiments:

  1. In the first one I used mRMR to get $81$ features, then random subsampling of $408$ observations, then Random Forest classification
  2. In the second one I first reduced the number of observations to $408$ with PCA (I could not do it after mRMR $81$-features selection because it would lead to only $81$ principal components), then I used mRMR to get $81$ features, finally Random Forest classification

At this point, my expectations were to see an increase of performance int the second experiment w.r.t. the first experiments. Instead, in the first experiment I get a macro-averaged ROC/AUC score of $92$%, while in the second I reach only $70$%.

Why the second experiment does not work as I expected?

  1. Peng, Hanchuan, Fuhui Long, and Chris Ding. "Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy." IEEE Transactions on pattern analysis and machine intelligence 27.8 (2005): 1226-1238.
  2. Ding, Chris, and Hanchuan Peng. "Minimum redundancy feature selection from microarray gene expression data." Journal of bioinformatics and computational biology 3.02 (2005): 185-205.
  3. Johnstone, Iain M., and D. Michael Titterington. "Statistical challenges of high-dimensional data." Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 367.1906 (2009): 4237-4253.
$\endgroup$
  • $\begingroup$ Have you considered using linear discriminant analysis (LDA), instead of PCA. PCA projects onto a hyperplane that preserves variation of the dataset, whereas LDA projects onto a hyperplane that maximizes component axes for class separation. sebastianraschka.com/Articles/2014_python_lda.html $\endgroup$ – John Yetter Aug 5 '16 at 12:44
  • $\begingroup$ Hi @JohnYetter, thanks for your comment. If I understood correctly LDA could be useful to maximize class separation when running dimensionality reduction on features. However, I am running PCA to find observations principal components instead of features PCs, so maybe I cannot introduce the class label as input for LDA. Did I understand correctly? $\endgroup$ – gc5 Aug 5 '16 at 13:05
  • $\begingroup$ If you want to select some "most representative observations", then PCA is not going to be the right tool. PCA is not selecting anything, it's constructing linear combinations. You might be better off with random subsampling. $\endgroup$ – amoeba Aug 5 '16 at 15:19
  • $\begingroup$ @amoeba Oh right, I got it.. Should I close it or do you want to put your comment as an answer? $\endgroup$ – gc5 Aug 5 '16 at 15:33
1
$\begingroup$

PCA cannot perform "subsampling" or selecting "most representative observations". It is not selecting anything, it is constructing linear combinations with maximum variance. In your case, PCA will be constructing linear combinations of observations, which has unclear meaning and hardly predictable consequences for your algorithm.

You should be better off with random subsampling.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.