Lets use some convention. Let
P be the number of features in your data,
N be the total number of examples.
mtry is the parameter in RF that determines the number of features you subsample from all of
P before you determine the best split.
nodesize is the parameter that determines the minimum number of nodes in your leaf nodes(i.e. you don't want to split a node beyond this).
When the parameter has a high range it is suggested using log scale. Why? You typically have larger jumps in accuracy(or what ever metric) when
mtry goes from 1 to 10 than when it goes from 90 to 100 and so testing all the numbers between 90 and 100 would waste more time. This is true of some other parameters as well. I would suggest watching this video as well as some of the resources linked by answers in this thread.
The second idea is using a modern hyper param search algorithm like Bayesian Optimization scheme here
Next you should get a feel for your data:
Do a small number of predictor variables have an outsized effect on response(higher
mtry may do better in this case). Or do a lot of small other variables matter as well(lower
mtry does better)? So you see this will effect the choice of your
mtry. Just fit a linear model to get a sense of feature importance. Maybe you will catch a lucky break and get a couple of features with really high importance and your model is simple.
If there is a lot of noise features then lower
mtry is not so useful. Scaltter plots are useful. You can also determine noise features by randomly permuting the X's of a single feature(i.e. just 1 column of the X) and comparing its fit before and after using a linear model against the Y. Noisy features will definitely not register any improvement in fit or somehow make it better after permutation (Word of Caution: Interactions between features may make this last step a bit dicey to use as a guiding tool to determine noise).
mtry ~ sqrt(P) is the rule of thumb based on academic paper linked in some of the answers here.
- You can do similar tests for other parameters to get an intuition(or positive confirmation) about your parameter values.
Breiman(2001) commented that the randomness used in the tree has to aim for low correlation while maintaining reasonable strength This is better unpacked in this https://arxiv.org/pdf/1804.03515.pdf paper that was linked by an answer above. I enjoyed reading it.