1
$\begingroup$

I am trying to fit a sklearn.ensemble.RandomForestClassifier. The [docs] explain that a matrix (rows - observations, columns - features). My observations are 700 000 short texts (tweets). My vocabulary is 1 000 000 unique words.

I tried doing mat = np.zeros((num_tweets, num_words)) but that resulted in an allocation of 5TB (num_tweets * num_words * 4 bytes).

What is the recommended solution? A sparse matrix? Generators? Something else?

$\endgroup$
1
$\begingroup$

Use sparse data types for sparse data

Today, English-language tweets have a character limit of 280 characters (previously, it was 140). This implies that a tweet can only contain a relatively small number of words, so a binary word-tweet matrix would have many 0 values. Using a sparse matrix will dramatically reduce your memory usage because you don't have to store all of the 0 values.

I'm not aware of a generator type that is compatible with sklearn classifiers. But the sklearn random forest classes do accept some sparse data types. For more information, see the documentation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.