I am trying to fit a
sklearn.ensemble.RandomForestClassifier. The [docs] explain that a matrix (rows - observations, columns - features). My observations are 700 000 short texts (tweets). My vocabulary is 1 000 000 unique words.
I tried doing
mat = np.zeros((num_tweets, num_words)) but that resulted in an allocation of 5TB (num_tweets * num_words * 4 bytes).
What is the recommended solution? A sparse matrix? Generators? Something else?