1
$\begingroup$

When testing my classifiers with new data, should I

  • remove stopwords from this data and/or
  • remove low information features from this data?

or just use the data the way it is?

In other words: Is removing stopwords and low information features only relevant for the data that I use to train and test the classifier or also for the data that the classifier will then classify?

Thanks!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

remove stopwords from this data

It depends on many things, such as the classifier and the features. E.g. it is quite common to remove stopwords when using bag-of-words, but very uncommon when using recurrent neural networks.

remove low information features from this data

People often use the validation set to explore feature selection.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your answer. I am using the bag of words model, so I should remove stopwords form the yet-to-classify data as well. I don't quite understand your other answer. I was not talking about the validation set, I was refering to completely new data. I have already trained and tested my classifier. $\endgroup$
    – user3813234
    Commented Oct 24, 2016 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.