2
$\begingroup$

The parameters required for a Random Forest classifier are as follows:

  • Depth, $d$
  • No. of random features, $K$
  • No. of trees, $I$
  • Randomizer seed, $R$

Which of the above are hyperparameters and which are model parameters?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ I think you should first look at CART tree parameters? Forest is a collection of trees. So all the parameters for trees should also good for forest. $\endgroup$
    – Haitao Du
    Jun 20, 2016 at 14:13

2 Answers 2

Reset to default
2
$\begingroup$

First, note that randomizer seed might be an argument in the implementation of Random Forest you're using, but it is not a parameter of the RF algorithm itself (it just sets the generator of pseudo-random numbers of the machine in a state so that the results are reproducible). Also, the abbreviations you're using ($I$, $K$, etc.) are not standard and seem to be specific to your implementation.

Anyhow, regardless of notation issues, the two main parameters of RF are the number of trees grown and the number of predictors randomly tried at each split.

What you call depth is sometimes found as the maximum node size, and controls the size of the trees that are grown. In the original implementation of RF, trees are grown to the maximum potential extent so that they reach the lowest possible bias. Then, variance is reduced by growing many trees and averaging them. The reason is that in RF, you can only decrease error by reducing the variance (where $error = bias + variance$), so the bias needs to be as low as possible in the first place. Therefore, in most cases, you don't really need to adjust this parameter (depth or max nodesize). Just make sure the default value allows the trees to grow as deep as possible.

Note that in some instances, it seems that slightly reducing the sizes of the trees can help in reducing overfitting (see the comments in that thread).

Finally, for machine learning algorithms such as RF, Boosting, etc., hyper-parameters and parameters are the same things (the proper name would be hyperparameters though). There is a slight semantic difference between the two when dealing with probability distributions. See for instance:

Also, somehow related:

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the intuition. Could you please point out the standard notations for the Random Forests? Also, just to be clear, is it wrong to call randomizer seeds as hyperparameters? They do affect the accuracy of the system nonetheless. $\endgroup$
    – Ébe Isaac
    Jun 15, 2016 at 9:58
  • $\begingroup$ there is no standard notation as far as I know, it varies depending on the implementation (R, Python, etc.). You could use the notation in the original RF paper, but it is not widely used. Anyway, the most important thing is to define your abbreviations at the beginning and then remain consistent throughout your code/study/paper. I don't know what is your implementation, but setting seeds should not have any impact on accuracy, it just ensures that the results are reproducible. $\endgroup$
    – Antoine
    Jun 15, 2016 at 10:06
  • $\begingroup$ I'm using WEKA through Python through Python-WEKA-Wrapper. At first I thought the same, but seeds actually do have an impact on the accuracy. Sometimes I see a change from 0.5% to even 3% just by adjusting the seed in Random Forest and AdaBoosting. $\endgroup$
    – Ébe Isaac
    Jun 15, 2016 at 10:12
  • 2
    $\begingroup$ if seed impact the accuracy, your model is not robust! $\endgroup$
    – Metariat
    Jun 15, 2016 at 11:59
  • $\begingroup$ @ÉbeIsaac this could plausibly be because accuracy is a volatile estimator of model quality. Check a proper scoring rule like log likelihood or Brier score. $\endgroup$
    – Sycorax ♦
    Jun 20, 2016 at 14:42
1
$\begingroup$
  • samplesize is also a typical hyperparameter. I determines how many random chosen samples are used for each tree.
  • replace is another one. You can sample observations for trees with replacement or without.
  • splitrule: in some packages like R-package randomForestSRC you can specify the splitting rule in the nodes.

Random Forests converge with growing number of trees, see Breiman, 2001, paper. So if you set you ntree very high (for small datasets (n<1000) 10000 should be enough) your results get more stable and the effect of the seed reduces. The number of trees should be set as high as possible (although in some special dataset constellations it could possibly reduce the accuracy, see here: Difference in randomForestSRC and randomForest package / increasing OOB-Error curve)

Improve this answer
$\endgroup$
2
  • $\begingroup$ Note that in the original RF, each tree is fitted to a bootstrap sample of the observations (as many observations as in the original sample drawn with replacement from the original sample). So I would not consider samplesize and replace to be real parameters, but more parameters that have been added for exploration purposes/flexibility in some implementations $\endgroup$
    – Antoine
    Jun 20, 2016 at 14:52
  • $\begingroup$ Well but "original RF" is not the bible and research is going on. I think these are parameters, that can improve the respective performance measure and are implemented in most of the implementations. I do not know what you mean with "real parameters". $\endgroup$
    – PhilippPro
    Jun 21, 2016 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.