How to tune hyperparameters in a random forest

Question

I don't know how I should tune the hyperparameters: "max depth" and "number of tree" of my model (a random forest).

I use Python and I just discovered grid search, but I don't know which range I should use at first.

Since my computer power is limited I can't just put a linear range from 0 to 100000 with a step of 10 for my two parameters. Is there any empirical recommendation/strategy to start with?

Answer 1

Number of trees is not a parameter that should be tuned, but just set large enough usually. There is no risk of overfitting in random forest with growing number of trees, as they are trained independently from each other. See our paper for more information about this: https://arxiv.org/abs/1705.05654

Max depth is a parameter that most of the times should be set as high as possible, but possibly better performance can be achieved by setting it lower.

There are more parameters in random forest that can be tuned, see here for a discussion: https://arxiv.org/pdf/1804.03515.pdf

If you have more than one parameter you can also try out random search and not grid search, see here good arguments for random search: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf

If you just want to tune this two parameters, I would set ntree to 1000 and try out different values of max_depth. You can evaluate your predictions by using the out-of-bag observations, that is much faster than cross-validation. ;)

Answer 2

Okay, So do max_depth = [5,10,15..], n_estimators = [10,20,30]. Separate your training set into a training' and validation' set. Then loop over all combinations of max_depth and n_estimators. For the most optimal loss on the validation' ,select that. Then you can finally calculate the validation error on the actual validation set.

You can simply just choose the optimal just based on the validation also,but you would be using the same set to evaluate again and again and due to multiple hypotheses testing it is likely you would get a better appearing solution on your validation but not on the final test.

Answer 3

Lets use some convention. Let P be the number of features in your data, X, and N be the total number of examples. mtry is the parameter in RF that determines the number of features you subsample from all of P before you determine the best split. nodesize is the parameter that determines the minimum number of nodes in your leaf nodes(i.e. you don't want to split a node beyond this).

When the parameter has a high range it is suggested using log scale. Why? You typically have larger jumps in accuracy(or what ever metric) when mtry goes from 1 to 10 than when it goes from 90 to 100 and so testing all the numbers between 90 and 100 would waste more time. This is true of some other parameters as well. I would suggest watching this video as well as some of the resources linked by answers in this thread.

The second idea is using a modern hyper param search algorithm like Bayesian Optimization scheme here

Next you should get a feel for your data:

1. Do a small number of predictor variables have an outsized effect on response(higher mtry may do better in this case). Or do a lot of small other variables matter as well(lower mtry does better)? So you see this will effect the choice of your mtry. Just fit a linear model to get a sense of feature importance. Maybe you will catch a lucky break and get a couple of features with really high importance and your model is simple.

2. If there is a lot of noise features then lower mtry is not so useful. Scaltter plots are useful. You can also determine noise features by randomly permuting the X's of a single feature(i.e. just 1 column of the X) and comparing its fit before and after using a linear model against the Y. Noisy features will definitely not register any improvement in fit or somehow make it better after permutation (Word of Caution: Interactions between features may make this last step a bit dicey to use as a guiding tool to determine noise).

mtry ~ sqrt(P) is the rule of thumb based on academic paper linked in some of the answers here.

3. You can do similar tests for other parameters to get an intuition(or positive confirmation) about your parameter values.

Breiman(2001) commented that the randomness used in the tree has to aim for low correlation while maintaining reasonable strength This is better unpacked in this https://arxiv.org/pdf/1804.03515.pdf paper that was linked by an answer above. I enjoyed reading it.