# Memory considerations with sklearn classfier

I am trying to fit a sklearn.ensemble.RandomForestClassifier. The [docs] explain that a matrix (rows - observations, columns - features). My observations are 700 000 short texts (tweets). My vocabulary is 1 000 000 unique words.

I tried doing mat = np.zeros((num_tweets, num_words)) but that resulted in an allocation of 5TB (num_tweets * num_words * 4 bytes).

What is the recommended solution? A sparse matrix? Generators? Something else?

I'm not aware of a generator type that is compatible with sklearn classifiers. But the sklearn random forest classes do accept some sparse data types. For more information, see the documentation.