RandomForestClassifier Parameter Optimization I'm a ML novice and I'm wondering if someone can critique what i'm doing (this is a bit open-ended).


*

*I have a very small corpus of text documents (n = 122). 

*There is a binary decision associated with each document. 

*I have created a "bag of words" representation of each document and I'm using python's RandomForestClassifier to make models to classify the data.


I'm tinkering with the parameters in the RandomForestClassifier in the following way:


*

*Run the RandomForestClassifer on 200 random subsets of the data (n = 112 for each of the 200 runs) (these #s were chosen arbitrarily).

*Rank the importance of each word in my bag of words matrix based on the average importance of each word in the 200 runs.


Now I want to see if there is an "optimal" # of feature/words for my data set using the RandomForestClassifier. 
This is done as follows:


*

*Generate 500 random forests (600 trees per forest). Each of the 500 forests uses 112 randomly chosen documents as a training set and the remaining 10 docs as a test set.

*Measure the average accuracy of these 500 forests as a function of # of words/features used to generate the models.


Here is what I see. The optimal average accuracy is around n=80 words/features.

Questions:


*

*I'm sure my approach is unorthodox. Is there a better way to optimize the RandomForest parameters?

*Is there any "intuitive" explanation for why my average accuracy seems to be optimal at around 80 words and then tails off? Is it simply that when n-features gets too large, my forests don't incorporate enough of the good features and so accuracy suffers?

*Any other parameters that are worth modifying here?

*Any other classification models worth looking at?


Thank you for any thoughts.
 A: To answer your second question, why accuracy tails off, I put together an example in R that should resemble your problem. I generated ~50 good predictors and ~1000 bad predictors (that are just randomly assigned dummy variables). I start by increasing the number of good predictors, and then after maxing those out I incrementally add in all of the bad predictors.
This illustrates what you observe in your data - up to a point the predictors are good and adding value, then at some point you're adding in the worse features and they start to drown out the good features.

The (admittedly messy) code is below:
library(data.table)
library(randomForest)
set.seed(343)
y <- sample(c(0,1), size=1500, replace=TRUE, prob=c(.8,.2))

pct_seq <- seq(.2,.1,by=-.002)

good.x <- sample(c(1,0), size=1500, replace=TRUE, prob=c(.21,.79))
for(i in pct_seq) {
    samp1 <- sample(c(1,0), size=1500, replace=TRUE, prob=c(i,1-i))
    samp0 <- sample(c(1,0), size=1500, replace=TRUE, prob=c(i/5,1-i/5))
    good.x <- cbind(good.x,ifelse(y==1,samp1, samp0))
}

pct_seq <- rep(.02,1000)

bad.x <- sample(c(1,0), size=1500, replace=TRUE, prob=c(.01,.99))
for(i in pct_seq) {
    samp1 <- sample(c(1,0), size=1500, replace=TRUE, prob=c(i,1-i))
    samp0 <- sample(c(1,0), size=1500, replace=TRUE, prob=c(i,1-i))
    bad.x <- cbind(bad.x,ifelse(y==1,samp1, samp0))
}

x <- cbind(good.x,bad.x)
y.fac <- as.factor(y)

var.seq <- c(seq(11,51, by=10), seq(151,951, by=100))
model.results <- data.frame(0,0)

for (j in var.seq) {
    print(j)
    print(randomForest(x[,1:j],y.fac,ntree=1000))
}

A: There is  few unorthodox (but not wrong) steps in your approach:
1) Usually, one does not use feature selection in sequence with classification. RF are usually used for one or for the other. It is not clear from the question whether you use the first step to select the "good" words and only use them in the second step, or not.
2) The three usual hyperparameters to set in RF are (in the order of importance - as I gather from peoples impression I do not know of any empirical research in this)


*

*the number of (randomly chosen) features to select in each tree construction step (max_features in sklearn, mtry in R)

*the number of trees per forest (n_estimators in sklearn, ntree in R)

*the maximum depth of the tree or dome measure of the size of the tree (here things diverge, sklearn limits the depth of the tree max_depth while R limits the size of the tree  nodesize and maxnodes)


You decided only to select the first hyperparameter, and not the others, which is ok, since it is considered as the most important (again, I know of no empirical evidence to that effect). 
3) You used way too many repetitions (500) to select the hyperparameter - my limited experience is that much less repetitions are needed (10) (but I do not have experience in text data - which is spare - many 0s). 
A: "Number of Features" parameter holds for the amount of randomness in the Random Forest (the fewer features you choose the more random your forest is). 
If you have lots of "relevant" features, you can choose small feature set to build each tree. But if only a fraction of your features is relevant, you better choose more features for each tree. In this case you can also perform feature selection before using Random Forest.
