Range of Values for Hyperparameter Fine-Tuning in Random Forest Classification

Question

I have implemented a random forest classifier. At the moment, I am thinking about how to tune the hyperparameters of the random forest. Of course, I am doing a gridsearch type of algorithm while checking CV errors. The problem is that I have no clue what range of the hyperparameters is even reasonable.

It is a binary classification problem, the dataset has 50 000 observations and 40 features.

I've read a paper that the RF algorithm is not able to overfit with respect to the number of trees. The more trees the better for the generalizability. Therefore, I thought I pick the number of trees fix and won't fine tune them. However, I am not sure what a reasonable number of trees would be. I guess I should consider the training data size and the number of features, but, I am still lost. Is 1000 enough?

For the subset selection of features at each node I think that I will try all values between 1 and 40. Would somebody suggest something different? I have seen that some people are just using the square root of the number of all features. However, I imagine that this might be problematic in some cases and that the correlation of the trees is not reduced sufficiently.

I control the individual tree depth by just setting a minimum number of observations in the leafs. Here I am completely lost, any suggestions? Maybe a series like 1, 5, 10, 20, 50, 100, 500, 1000, 2000, 4000, 8000, 16000, 32000? Would you control for the individual tree depth with another hyperparameter as well? Max tree levels?

Any suggestions would be helpful. I appreciate every opinion!:)

Answer 1

you are right that the random forest or other tree ensemble methods make it hard for overfitting. Essentially, you can set the number of trees to be very large, it is uncommon to have 5000 trees. The more trees you have, the less variance there will be in your result. So this parameter is something you can just be less worried about, go for an arbitrarily large enough number and normally you will get good result.

However, you should not grow the tree too deep. Otherwise, you end up having just very few examples in each node, this will likely make your training error extremely low and your test error a bit higher than you would except. Depends on the library you are using, normally there is a default number of minimum end nodes in the leaf. You can increase this number and see how it affects the result on the validation set. For example, in R, the default number is 5. Then you can increase this number until you observe that the validation error starts to go down. Maybe this happens already when you have minimum end nodes equals 10 or so (depending on the data you have).

Answer 2

The usual hyperparameters for a RF are

• number of trees
• number of attributes that are randomly selected for the split search when constructing each tree
• limits on the tree depth (for each tree)

in sklearn https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html these are the n_estimators, max_features and max_depth

in the R package randomForest https://www.rdocumentation.org/packages/randomForest/versions/4.6-14/topics/randomForest they are ntree, mtry, and there is no equivalent to the max tree depth.

The number of trees is usually not important and could be fixed if the number of high. If you have no concerns regarding computation time, ser the number of trees to 1000 and I believe, there is no need to search for other values. If you do, the search should be geometric like 100, 200, 500, 1000, 2000, and so on.

The number of features to select is by default the square root of the number of atributes (for classification) or the number of attributes for regression. In my experience, this hyperparameter is not that important and if you have limits on the time to do the hyperparameter search, you can accept the default. If you want to search, in your case test for 6 ,7 10, 12 and maybe 20 (for classification)

The last hyperparameter (limits of the tree depth) is also not significant, in my experience. The randomForest package, controls the depth by the minimum number of cases to perform a split in the tree construction algorithm, and for classification they suggest 1, that is no constraints on the depth of the tree. Sklearn uses 2 as this min_samples_split. If you plan to search this hyperparameter, I think it is wiser to control the minimum number of samples to split the tree, and 1, 2 or 5 seems reasonable values.

To summarize, in my experience the defaults for the RF hyperparameters are usually good enough (provided ntree is large - I think sklearn default of 100 trees is too low - it was even lower in previous versions of the package). This is the main advanatge of RF - usually you do not need to search for hyperparameters and it is trivially parallelizable if training time is a problem, and it is likely on of the three best algorithms for most classification problems (together with RBF SVM and gradient boosting).