Classification performance and the feature set selection I am now working on a classification problem. The generated feature set can be separated into two group. I did a comparison study: use all of the features; use the features of group 1 only; and use the features of group 2 only. It turns out that the performance of using all of the features are the worst. How to understand this kind of observation?
 A: The symptoms you are describing sound like a classic case of overfitting.  As described in the link, over time, researchers have managed to come up with various strategies such as pruning or early stopping to try to avoid or reduce the effects of overfitting.  Depending on which specific learning algorithm you are using, and the details of how it was implemented, it may be more or less sensitive to this problem.  If you are using a machine learning package such as scikit-learn or Weka, I'd suggest swapping out whatever learning algorithm you were previously using, and try running the same experiment with a different algorithm; both packages make that extremely easy (almost trivial, in fact) to do.
A: I think this is highly dependent on the classification approach that you chose.  Many classifiers can get stuck in local optima that represent non-ideal parameter spaces.  Another possibility is the scale of your normalization may be different between the feature sets thus causing issues with the classification.
A: I would suggest you do a cross validation to observe the influence of feature numbers. You need to quantify both in-sample error (in training data) and out-of-sample error (cross validation error). If both are large, it is possible due to the underfitting (high bias), and you need to add more features; if the in sample error converges to a small value, but the cross validation error is unacceptably high, you need to remove some features to reduce the possible overfitting (high variance). 
