I have been using many approaches like the random forest, ANOVA, chi^2, etc. to select the most important features for machine learning. After I read more on the feature selection and dimensionality reduction, I find that there is no one method that fits all cases. I find that many books about machine learning mention the principal component analysis. I find a dataset with 60 features (independent variables) and 1 dependent variable, I apply the PCA on it and find that 8 features' variance contributing to 95% of the total variance. As many books explain, PCA is an unsupervised method, while most ML-based feature selection methods are supervised approaches. In this sense, I do not see how PCA could help to reduce the dimensionality (by removing those low-variance variables). I find one thing very interesting. I apply a random forest method and PCA on the same datasets, which turns out that the most important features from the random forest are totally different from those given my PCA. I prepare two data sets, one with variables, suggested by random forest, removed and one with variables suggested by PCA, removed, and then fit them into the same regression model with the machine learning method. The one with the random forest has an accuracy of 20% higher than that on PCA. I am not sure if the result makes sense or not, but if it is true, how could one use PCA for features selection?
1 Answer
You mention it in the question, but this is basically the difference between supervised and unsupervised feature selection methods. Supervised methods select features that are best for a particular task, usually for discriminating between groups of samples. With unsupervised methods, you have no knowledge of any groups in the data, so you just want to select features that capture the most variability. There's no particular reason to expect that the variables that capture the most variance will also be the best for discriminating between particular groups, or vice versa. If you already know the groups you're trying to describe with fewer features, supervised methods will likely be preferable.
Do be careful to not bias your model output by performing feature selection too early, though - you should split the data into train/test cohorts, select features on the training data, train your model on the training data, and finally evaluate on the test data. Do not perform feature selection and then split into train/test, or evaluate accuracy/other metrics on the training data. The way you describe your training procedure, it sounds like there is some bias in one or both of the feature selection or evaluation.
