Question: when you have create a corpus of let’s say, 10,000 documents, and the vocabulary size made for these is let’s say, 1 million, what best practices exist to either work with this type of document matrix or is there a better way to be smart about the max_features for a vocab matrix?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Depends what you want to do with it. If you want to extract topics from the documents, gensim provides ways to process the data in a very memory efficient way. It is good practice however to preprocess your text and clean your tokens for stop words etc before analysing it.
