73
$\begingroup$

My questions are about Random Forests. The concept of this beautiful classifier is clear to me, but still there are a lot of practical usage questions. Unfortunately, I failed to find any practical guide to RF (I've been searching for something like "A Practical Guide for Training Restricted Boltzman Machines" by Geoffrey Hinton, but for Random Forests!

How can one tune RF in practice?

Is it true that bigger number of trees is always better? Is there a reasonable limit (except comp. capacity of course) on increasing number of trees and how to estimate it for given dataset?

What about depth of the trees? How to choose the reasonable one? Is there a sense in experimenting with trees of different length in one forest and what is the guidance for that?

Are there any other parameters worth looking at when training RF? Algos for building individual trees may be?

When they say RF are resistant to overfitting, how true is that?

I'll appreciate any answers and/or links to guides or articles that I might have missed while my search.

$\endgroup$
1
49
$\begingroup$

I'm not an authoritative figure, so consider these brief practitioner notes:

More trees is always better with diminishing returns. Deeper trees are almost always better subject to requiring more trees for similar performance.

The above two points are directly a result of the bias-variance tradeoff. Deeper trees reduces the bias; more trees reduces the variance.

The most important hyper-parameter is how many features to test for each split. The more useless features there are, the more features you should try. This needs tuned. You can sort of tune it via OOB estimates if you just want to know your performance on your training data and there is no twinning (~repeated measures). Even though this is the most important parameter, it's optimum is still usually fairly close to the original suggest defaults (sqrt(p) or (p/3) for classification/regression).

Fairly recent research shows you don't even need to do exhaustive split searches inside a feature to get good performance. Just try a few cut points for each selected feature and move on. This makes training even faster. (~Extremely Random Forests/Trees).

$\endgroup$
4
  • $\begingroup$ A couple other notes: in practice I usually confirm convergence by comparing predictions from one half of the trees to the other. As far as overfitting, it's more of a function of what you're trying to generalize to. They aren't going to overfit if you are training on a representative sample, but that's seldom how it really works. $\endgroup$ – Shea Parkes Mar 25 '13 at 17:34
  • $\begingroup$ Is your 'deeper trees = better, all else constant' true for extremely noisy data with dependence structures that change over time, in which linear relationships are the most robust to not change between the training set and test set? $\endgroup$ – Jase Jan 30 '14 at 10:28
  • $\begingroup$ I could see potential for shallower trees to be better if you have a situation where you should only learn shallow relationships, but I'd really want to use empirical evidence to prove it (and have no time to work on that). If you believe or have proof that the linear relationships are the most resilient, then I would strongly consider something not tree-based. Maybe neural networks with skip layers? $\endgroup$ – Shea Parkes Feb 1 '14 at 1:59
  • $\begingroup$ Well lets say that you have a dataset with 3 relevant features and 100 features that are white noise, and 50 datapoints. But you don't know which are white noise and which are relevant ahead of time, you just know that are your data is so noise that this is the case. Clearly extremely shallow trees with a large mtry is better, no proof or empirical evidence is needed to see this. $\endgroup$ – Jase Feb 1 '14 at 6:30
23
$\begingroup$
  • Number of trees: the bigger the better: yes. One way to evaluate and know when to stop is to monitor your error rate while building your forest (or any other evaluation criteria you could use) and detect when it converges. You could do that on the learning set itself or, if available, on an independent test set. Also, it has to be noted that the number of test nodes in your trees is upper bounded by the number of objects, so if you have lots of variables and not so many training objects, larger forest will be highly recommended in order to increase the chances of evaluating all the descriptors at least once in your forest.

  • Tree depth: there are several ways to control how deep your trees are (limit the maximum depth, limit the number of nodes, limit the number of objects required to split, stop splitting if the split does not sufficiently improves the fit,...). Most of the time, it is recommended to prune (limit the depth of) the trees if you are dealing with noisy data. Finally, you can use your fully developed trees to compute performance of shorter trees as these are a "subset" of the fully developed ones.

  • How many features to test at each node: cross-validate your experiences with a wide range of values (including the recommended ones), you should obtain a performance curve and be able to identify a maximum pointing out what is the best value for this parameter + Shea Parkes answer.

  • Shea Parkes mentionned the Extra-Trees, here is the original paper describing in details the method: http://orbi.ulg.ac.be/bitstream/2268/9357/1/geurts-mlj-advance.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.