4
$\begingroup$

I'm trying to make decisions regarding Random forest parameters for classification. My dataset contains 26 features and 6300 instances. How can I decide the values of (the number of trees, number of features, number of the bootstrap sample and the minimum node size) is there any formulas I should calculate help to decide about these values?

Thank you

$\endgroup$
  • 5
    $\begingroup$ Random forest is nice because it does well using rules-of-tumb and without requiring much parameter tuning. But most people still find it worthwhile to do hyperparameter optimization with RF; there are many, many threads on strategies to do this with key terms such as cross-validation, grid search, and random search. $\endgroup$ – Sycorax says Reinstate Monica Mar 30 '18 at 17:36
  • $\begingroup$ @Sycorax Thank you very much for your comment. Since I'm using weka unfortunately I didn't find a good resource of how to optimize random forest with grid search or any other algorithm could serve the job, either using the GUI or with java. $\endgroup$ – User505 Mar 30 '18 at 18:52
  • $\begingroup$ Sure, I understand that Weka is somewhat limited in terms of its offerings. By the same token, that's a compelling reason to, in the long-term, build familiarity with a more sophisticated tool. $\endgroup$ – Sycorax says Reinstate Monica Mar 30 '18 at 19:01
4
$\begingroup$

Read the first RF article for implementation of RF in the R-project ("R-News"). I commonly use 500 to 5000 trees, but Breiman recommend using more in his original Machine Learning paper (he said: "Don't be stingy").

I have written the following for my own codes:

Number of features used for training at each node split, jtry. A unique characteristic of RF is that only a small number of features are randomly selected for training each tree. The parameter used to specify the number of features used is jtry, and a typical value is jtry=$\sqrt{p}$, where $p$ is the total number of features. Small values of jtry of 1 and 2 have shown to yield high performance values. If there are a large number of features to filter, then larger values of jtry will be required. During tree node splitting, jtry features are randomly selected from the $p$ features, and used to create node splits.

Number of trees, ntree. The number of trees used is specified by ntree. It has been shown that as few as 50 trees can result in reliable results (Cutler et al, 2007), but the majority of applications commonly use 500, 1000, 2000, or more trees. Breiman has recommended using at least 1000 trees (Breiman, 2001); however, it is common to use 500 trees, unless of course more trees results in greater performance.

Number of objects in terminal nodes, nodesize. The parameter nodesize controls the size of terminals nodes during node splitting while training a tree. Nodes with fewer than nodesize objects are not split, and therefore become terminal nodes. The ability of a tree to generate terminal nodes with class purity depends on the quality of the jtry features randomly selected for making each split as well as the value of nodesize. If nodesize$=1$, however, then every terminal node will have class purity.

Number of total nodes, nnode. The number of total nodes grown in a tree can also be limited by use of the parameter nnode. Features that are less informative for splitting objects according to class labels will result in trees with greater size, and therefore setting the upper bound to 500 or 1000 is certainly not unreasonable if there are hundreds of samples.

L. Breiman. Random Forests. Machine Learning, 45:5-32, 2001.

D.R. Cutler, T.C. Edwards, Jr., K.H. Beard, A. Cutler, K.T. Hess, J. Gibson, J.J. Lawler. Random forests for classification in ecology. Ecology, 88(11):2783-2792, 2007.

$\endgroup$
  • 1
    $\begingroup$ Thank you very much for your answer. Do you have any papers that mention such suggestion. $\endgroup$ – User505 Mar 30 '18 at 17:18
  • 1
    $\begingroup$ These suggestions go back to Breiman's original paper: stat.berkeley.edu/users/breiman/randomforest2001.pdf $\endgroup$ – Matthew Drury Mar 30 '18 at 17:35
  • 1
    $\begingroup$ I modified the original post, with citations. Adele Cutler, in the second citation was one of Breiman's doctoral students, at UC-Berkeley. $\endgroup$ – JoleT Mar 30 '18 at 17:52
  • 1
    $\begingroup$ Thank you very much for your valuable answer. I believe you made this question a good reference for many researchers! Thanks again $\endgroup$ – User505 Mar 30 '18 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.