Can we improve a model by dropping unimportant features

Question

I have a Random Forest Model which, after using StringIndexer and HotEncoder, has got around 1300 features.

I calculated the importance of all these features and I found out that more than 500 features have got 0 importance in the final model.

Will I improve my model by dropping this feature before the training or will I end up just with a model faster to train?

Since those unimportant features are mostly features generated by the hotencoder for categories which almost never appear (those which belong to the "long tail" of the dataset) I was considering as an option to group them into a new feature named "other" before start the training the model.

Answer 1

Rare categoricial values have indeed no chance of being chosen in a good decision tree. Mapping them to "other" can be worth a try, but I wouldn't expect this to improve results much unless you have reason to believe that "being other" is predictive if your class label (e.g., predicting rare classes).

Dropping attributes that were never chosen by a random forest clearly shouldn't affect the quality of the classifier, but improve training time.

Answer 2

applying algorithms like one hot encoding will definitely bloat the feature count. Say, the original featureset is X containing features x1,x2,...,xn. The focus should be on determining relevant features before applying a encoding algorithm. This way, you'll know if features x1,x2,...,xn are important or not. In clustering you can apply, principal component analysis or multiple factor analysis to determine the relevant features for the model. Similar algorithms exist for classification.