0
$\begingroup$

I have a Naive-Bayes model that does sentiment analysis. For the training of the model, the training data was cleaned, i.e.: stop words were removed, certain punctuation, etc...

When I want to classify as new sentence, say "I love CrossValidated; it is my favourite site!", should I be "cleaning" it before giving it to the classifier?

$\endgroup$
2
$\begingroup$

Yes, you should pre-process the test data (that is, the new input) in the same way. The classifier is trained to classify data without punctuation and so on; that is the form of data that it expects as input. Generally, one should pre-process test data in the same way that one pre-processes training data.

There are exceptions to this generic principle, however. In certain applications, it is common to perturb (add noise to) replicated training data. This is done in order to make the classifier output smoother and less "jittery" when exposed to new data. Test data is usually not perturbed in the same way.

$\endgroup$
  • $\begingroup$ Thanks. How is this perturbation done; is the training data artificially injected with random words here and there? $\endgroup$ – turnip Feb 6 '18 at 12:29
  • $\begingroup$ I'm not sure whether it is common to do this specifically with text data, but I suppose that one way to do this is to replace words with synonyms. $\endgroup$ – Martin L Feb 7 '18 at 12:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.