I'm creating a news classifier using the reuters dataset. Right now I'm in the process of preparing the dataset for training. First I removed all punctuation, numbers and special characters and after a created a list with all the unique words in every piece of news. To improve training I want to remove neutral words that won't affect classification, like "the", "of", "a" , etc. What I tried doing is, get the ratio of the ocurrences of one word in one text with the total number of words in that particular text, then to that for every text and take the average of those ratios and then choose a certain threshold to remove a word from the vocabulary, but this way it takes too long. Is there a better way? This is a multi-class problem.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Two simple things people commonly do is:
- Use one of the many available lists of stopwords and filler them out first. Most NLP software does this out-of-the-box.
- Decide on the number of words that will fit your memory (considering also the memory that will be available when the model is deployed, so e.g. if this is going to be mobile app, this needs to be adequately small) and take the $N$ most common words.
But you don’t have to filter out anything, you can use hashing trick to loosely map the words to lower dimensionality.
