I'm a ML novice and I'm wondering if someone can critique what i'm doing (this is a bit open-ended).
- I have a very small corpus of text documents (n = 122).
- There is a binary decision associated with each document.
- I have created a "bag of words" representation of each document and I'm using python's
RandomForestClassifierto make models to classify the data.
I'm tinkering with the parameters in the
RandomForestClassifier in the following way:
- Run the
RandomForestClassiferon 200 random subsets of the data (n = 112 for each of the 200 runs) (these #s were chosen arbitrarily).
- Rank the importance of each word in my bag of words matrix based on the average importance of each word in the 200 runs.
Now I want to see if there is an "optimal" # of feature/words for my data set using the RandomForestClassifier. This is done as follows:
- Generate 500 random forests (600 trees per forest). Each of the 500 forests uses 112 randomly chosen documents as a training set and the remaining 10 docs as a test set.
- Measure the average accuracy of these 500 forests as a function of # of words/features used to generate the models.
Here is what I see. The optimal average accuracy is around n=80 words/features.
- I'm sure my approach is unorthodox. Is there a better way to optimize the RandomForest parameters?
- Is there any "intuitive" explanation for why my average accuracy seems to be optimal at around 80 words and then tails off? Is it simply that when n-features gets too large, my forests don't incorporate enough of the good features and so accuracy suffers?
- Any other parameters that are worth modifying here?
- Any other classification models worth looking at?
Thank you for any thoughts.