When testing my classifiers with new data, should I

  • remove stopwords from this data and/or
  • remove low information features from this data?

or just use the data the way it is?

In other words: Is removing stopwords and low information features only relevant for the data that I use to train and test the classifier or also for the data that the classifier will then classify?



remove stopwords from this data

It depends on many things, such as the classifier and the features. E.g. it is quite common to remove stopwords when using bag-of-words, but very uncommon when using recurrent neural networks.

remove low information features from this data

People often use the validation set to explore feature selection.

  • $\begingroup$ Thanks for your answer. I am using the bag of words model, so I should remove stopwords form the yet-to-classify data as well. I don't quite understand your other answer. I was not talking about the validation set, I was refering to completely new data. I have already trained and tested my classifier. $\endgroup$ – user3813234 Oct 24 '16 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.