3
$\begingroup$

I'm trying to make decisions regarding Random forest parameters for classification. My dataset contains 26 features and 6300 instances. How can I decide the values of (the number of trees, number of features, number of the bootstrap sample and the minimum node size) is there any formulas I should calculate help to decide about these values?

Thank you

$\endgroup$
3
  • 5
    $\begingroup$ Random forest is nice because it does well using rules-of-tumb and without requiring much parameter tuning. But most people still find it worthwhile to do hyperparameter optimization with RF; there are many, many threads on strategies to do this with key terms such as cross-validation, grid search, and random search. $\endgroup$
    – Sycorax
    Mar 30, 2018 at 17:36
  • $\begingroup$ @Sycorax Thank you very much for your comment. Since I'm using weka unfortunately I didn't find a good resource of how to optimize random forest with grid search or any other algorithm could serve the job, either using the GUI or with java. $\endgroup$
    – User505
    Mar 30, 2018 at 18:52
  • $\begingroup$ Sure, I understand that Weka is somewhat limited in terms of its offerings. By the same token, that's a compelling reason to, in the long-term, build familiarity with a more sophisticated tool. $\endgroup$
    – Sycorax
    Mar 30, 2018 at 19:01

1 Answer 1

4
$\begingroup$

Read the first RF article for implementation of RF in the R-project ("R-News"). I commonly use 500 to 5000 trees, but Breiman recommend using more in his original Machine Learning paper (he said: "Don't be stingy").

I have written the following for my own codes:

Number of features used for training at each node split, jtry. A unique characteristic of RF is that only a small number of features are randomly selected for training each tree. The parameter used to specify the number of features used is jtry, and a typical value is jtry=$\sqrt{p}$, where $p$ is the total number of features. Small values of jtry of 1 and 2 have shown to yield high performance values. If there are a large number of features to filter, then larger values of jtry will be required. During tree node splitting, jtry features are randomly selected from the $p$ features, and used to create node splits.

Number of trees, ntree. The number of trees used is specified by ntree. It has been shown that as few as 50 trees can result in reliable results (Cutler et al, 2007), but the majority of applications commonly use 500, 1000, 2000, or more trees. Breiman has recommended using at least 1000 trees (Breiman, 2001); however, it is common to use 500 trees, unless of course more trees results in greater performance.

Number of objects in terminal nodes, nodesize. The parameter nodesize controls the size of terminals nodes during node splitting while training a tree. Nodes with fewer than nodesize objects are not split, and therefore become terminal nodes. The ability of a tree to generate terminal nodes with class purity depends on the quality of the jtry features randomly selected for making each split as well as the value of nodesize. If nodesize$=1$, however, then every terminal node will have class purity.

Number of total nodes, nnode. The number of total nodes grown in a tree can also be limited by use of the parameter nnode. Features that are less informative for splitting objects according to class labels will result in trees with greater size, and therefore setting the upper bound to 500 or 1000 is certainly not unreasonable if there are hundreds of samples.

L. Breiman. Random Forests. Machine Learning, 45:5-32, 2001.

D.R. Cutler, T.C. Edwards, Jr., K.H. Beard, A. Cutler, K.T. Hess, J. Gibson, J.J. Lawler. Random forests for classification in ecology. Ecology, 88(11):2783-2792, 2007.

$\endgroup$
5
  • 1
    $\begingroup$ Thank you very much for your answer. Do you have any papers that mention such suggestion. $\endgroup$
    – User505
    Mar 30, 2018 at 17:18
  • 1
    $\begingroup$ These suggestions go back to Breiman's original paper: stat.berkeley.edu/users/breiman/randomforest2001.pdf $\endgroup$ Mar 30, 2018 at 17:35
  • 1
    $\begingroup$ I modified the original post, with citations. Adele Cutler, in the second citation was one of Breiman's doctoral students, at UC-Berkeley. $\endgroup$
    – user32398
    Mar 30, 2018 at 17:52
  • 1
    $\begingroup$ Thank you very much for your valuable answer. I believe you made this question a good reference for many researchers! Thanks again $\endgroup$
    – User505
    Mar 30, 2018 at 18:44
  • $\begingroup$ Just wanted to mention that jtry is also the same asmtry if anyone is searching for examples or other literature in R! $\endgroup$
    – PleaseHelp
    Mar 18, 2021 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.