I'd like to design a feature extraction, selection, and classification scheme to use on novel data sets. For each row in a table I calculate 10 features. I then select which features are relevant (using a training set), and finally predict a label for each row using a Naive Bayes classifier (using a testing set). Ideally, this would be automated such that a user can just load a table and click "go".
Here's my issue. Even with feature selection, classification using a single, known relevant feature can outperform classification on all features. That is, running the classifier using a single feature that we know performs well can yield better predictions than running the classifier on all selected relevant features. I know that classifiers are "garbage in, garbage out", so including irrelevant features can lower the performance. But I thought the feature selector would prevent garbage from going into the classifier.
Am I maybe not being strict enough with my feature selection? Should I include less features in the first place? Is there an obvious error I'm making somewhere else?
Here's a breakdown of what I'm doing in case it's useful.
- Calculate 10 features for each row of the table.
- Split the table into testing and training in a k-fold scheme.
- Select the relevant features using the training data and the criterion
fisher-ratio > 0.2. For this I used the function IndFeat.m in matlab, which calculates fisher's ratio between the training labels and the feature of interest. A higher ratio means the feature is more informative about the labels. A description of fisher's ratio is here, and a description of the matlab function is here.
- Using the selected features, classify on the testing data.
- Repeat steps 2-5 for each k folds.