I'm working on a dataset with 200,000+ samples and approximately 50 features per sample: 10 continuous variables and the other ~40 are categorical variables (countries, languages, scientific fields etc.). For these categorical variables, you have for example 150 different countries, 50 languages, 50 scientific fields etc...
So far my approach is:
For each categorical variable with many possible value, take only the one having more than 10000 sample that takes this value. This reduces to 5-10 categories instead of 150.
Build dummy variable for each categorical one (if 10 countries then for each sample add a binary vector of size 10).
Feed a random forest classifier (cross-validate the parameters etc...) with this data.
Currently with this approach, I only manage to get 65% accuracy and I feel like more can be done. Especially I'm not satisfied with my 1) since I feel like I shouldn't arbitrarily remove the "least relevant values" according the the number of sample they have, because these less represented values could be more discriminative. On the other hand, my RAM can't afford adding 500 columns * 200000 rows to the data by keeping all possible values.
Would you have any suggestion to cope with this much categorical variables?