2
$\begingroup$

As far as I understand the splitrule = “extratrees” option in the package ranger is an implementation of Geurts et al (2006) extremely randomized trees.

In their paper they state:

At each tree node, this is combined with a random choice of a certain number of attributes among which the best one is determined. In the extreme case, the method randomly picks a single attribute and cut-point at each node, and hence builds totally randomized trees whose structures are independent of the target variable values of the learning sample.

  1. If I run ranger with splitrule = "extratrees" but without specifying mtry I see in the resulting object that mtry = 8. Doesn’t the algorithm also imply a random choice of mtry as stated in the article?

  2. On Wikipedia there is a small section about extremely randomized trees. In contrast to the paper, it does not mention that the number of features chosen is random but can be specified, which would be in line with what I see in ranger. Then I wonder: Could I then still call this extremely randomized trees as implemented in ranger according to Geurts et al when writing a paper or is this a "mix" of the default and extremely randomized trees?

  3. Wikipedia furthermore says:

    first, each tree is trained using the whole learning sample (rather than a bootstrap sample)

Nevertheless, I get the out of bag error in the fit object of ranger, which in my understanding is impossible if the whole training data is used. Only a cross-validation error would be possible unless in fact bagging is still performed.

So, basically my question is: What exactly is implemented in ranger under the option splitrule = "extratrees" and why did they deviate from the original paper?

$\endgroup$
3
$\begingroup$

Your understanding is correct, extraTrees does implement Geurts et al. (2006) Extremely randomized trees. The implementation of extraTrees has been discussed in detail in Github's thread on that issue so I would strongly urge you read it further here.

Regarding the particular questions raised:

  1. Yes, but read on! By default the same number of candidates mtry is used, this being calculated as the (rounded down) square root of the number variables. It makes sense for ranger to use a particular number for mtry because that way it can be effectively regularised. Theoretically, we should indeed restrict the choice to a random subset to begin within each tree. Nevertheless as the choice of the attribute to split as well as the split itself are random this difference is mostly a formality.
  2. Yes, you can. By far the main novelty in Geurts et al. is the way that nodes are split by choosing cut-points fully at random; that is something that ranger definitely does. You could/should specify in the paper that the implementation used is the one from ranger to alleviate any uncertainties.
  3. You are correct; the default values hurt us here. That said if during training, we sample without replacement (i.e. we set replace = FALSE) as well as set the fractions of observation to sample to 1 (i.e. we set sample.fraction = 1) we will not get a OOB error and the forest is trained on the whole sample. You might want to create a new issue about this in ranger's github repo. It will mostly be a case of re-adjusting the defaults when splitrule='extraTrees' but we can do it manually too.

To recap: If we use rf <- ranger( ..., splitrule = "extratrees", replace = FALSE, sample.fraction = 1) we can safely say that we use Geurts et al. implementation. All the core differences between extremely randomized trees and "standard" random forests are respected.

Minor point on regarding reporting OOB or not: While indeed OOB estimates can substitute the presence of a separate test set, I find it much more clear if there is a distinct test set and/or we use repeated CV or bootstrapping. It makes the comparison with other approaches (e.g. a simple linear model or an SVM) more comparable and coherent.

$\endgroup$
  • 2
    $\begingroup$ I think we should add a note to the docs of ranger but don't change the defaults (the other parameters shouldn't change when just selecting another split rule). I created an issue for that: github.com/imbs-hl/ranger/issues/408 $\endgroup$ – mnwright May 7 at 7:52
  • $\begingroup$ @mnwright: You are the boss Marvin. I am just a happy ranger user. $\endgroup$ – usεr11852 says Reinstate Monic May 7 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.