May using more features decrease the accuracy of a classifier?

Question

I'm testing a project. I have train and test data. There are 182 features and 1000 samples for train and 3500 for test. If I select certain columns of data and apply naive Bayes classifier to them, I have a higher accuracy than when I use the whole 182 features.

For example, for the following example for num=30 the accuracy is about 0.75, for 100 it's about 0.60 and for 182 it's about 0.44!!!

How is it possible? I thought using more features leads to a better performance.

num =182
naive_bayes.fit(x_train[:,range(num)], y_train)
y_pred = naive_bayes.predict(x_test[:,range(num)])
acc = accuracy_score(y_test, y_pred)   
print("acc:", acc)

And it's the plot of accuracy per the number of features.

Answer 1

This is an instructive encounter with Hughes phenomenon. Naïvely, one would think that the more information one has the better one can model a system and make predictions. However, this prejudice ignores the so-called curse of dimensionality.

Suppose for convenience that each feature (or variable) can only take on a finite number of values. In order to capture the characteristics of the data accurately, one first needs a large enough collection of data points that, so to speak, fills the feature space, i.e. one needs enough samples with each combination of values. Now, in practice, when you are presented with the data, you have a limited number of observations. If you have too many features, the feature space will have so many subregions with very few observations or none at all that your classifier will lose predictive power given that it did not get to learn the behaviour of the data in too many significant subregions.

This problem typically does not occur when you have a small number of variables because very few possible combinations are sharing the limited number of observations.

Whilst raising the number of features was helpful for the predictive power of the classifier initially, it became a liability once it meant that adding features would prevent the classifier from learning the behaviour of the data in too many sizeable regions. The classifier runs the risk of overfitting the data in the data-heavy observed regions.

The reasoning naturally extends to the case with continuous features.

This article might help: https://towardsdatascience.com/the-curse-of-dimensionality-50dc6e49aa1e.

Answer 2

Adding too many predictors will lead to overfitting. Always. Take a look at our overfitting tag.

Don't just throw predictors into your model. (Cross-validation and regularization help somewhat, but they will not prevent all overfitting.)

Also: Why is accuracy not the best measure for assessing classification models?

Answer 3

This has been answered but here are a few more tips. When creating a model, you must always be conscious of the bias/variance trade off curves of the model. If a model has many features, it will predict with high variance, causing less accurate results. Too few features, and the model will have high bias, causing the model too often predict near the same value, which can also reduce accuracy. It's important to use proper feature engineering in order to find the optimum tradeoff between bias and variance.