The performance of the classifier went bad after adding extra two features I'm solving a classification problem using 10 features and logistic regression. The performance of the classifier is fine when I use the 10 features only, however when adding another 2 features, the performance gets a little worse. 
When I use the 2 features alone with the classifier I get a performance as good as using the 10 features. 
Now my question is why would the performance gets worse when I use the 12 features combined?! Since using the 10 features with the classifier alone and the 2 features with the classifier alone gave almost identical performance, then why combining them would cause a problem? I would expect the performance to increase or at least be the same!
Edit:
is it possible that this is a 'curse of dimensionality'?
 A: The issue of having too many variables in your model is called over-fitting. In predictive modeling, the idea is to use the data at hand to discover the trends that exist and that can be generalized to future data. By including variables in your model that have some minor, non-significant effect you are abandoning this idea. What you are doing is considering the specific trends in your specific sample that are only there because of random noise instead of a true, underlying trend. In other words, a model with too many variables fits the noise rather than discovering the signal.
As an exaggerated example, consider a model where $\hat{Y}_i=Y_i$. In other words, the predicted value equals the observed value. In this example we have an extreme case of over-fitting (in fact, we have a perfect fit). However, we are not discovering the general trends that the data is generated from.
When you have more variables in the model, $SSE=\sum_i^n(Y_i-\hat{Y}_i)^2$ is necessarily decreasing which means, on average, you are getting closer to the exaggerated $\hat{Y}_i=Y_i$ example. 
It might seem weird that we want our predicted value to be close to the observed value, but not too close. This is why many people like cross-validation techniques to help with model selection and it's one of the reasons why some people will say "statistics is an art not a science".
