I am having trouble with the intuition for running several RF models.

I have a few features (between 3 and 10) which should be correlated, since they measure things that are usually related.

I want to tune the maximum depth of the tree, and the min samples at each leaf -both of which are used as stopping criteria.

Since the data is correlated, my best intuition is that I would want to make each decision tree as deep as possible, and err on the side of a few min samples at each leaf (let's say 10, given that there are only about 1000 data points). My justification for this is that there is little concern for over-fitting since the data is correlated.

Is this intuition correct? And if not, what is a good way to optimize these two parameters?


You are doing it wrong -- the essential part of RF is that it basically only requires making # trees large enough to converge and that's it (it becomes obvious once one starts doing proper tuning, i.e. nested cross-validation to check how robust the selection of parameters really is). If the performance is bad it is better to fix the features or look for an other method.

Pruning trees works nice for decision trees because it removes noise, but doing this within RF kills bagging which relays on it for having uncorrelated members during voting. Max depth is usually only a technical parameter to avoid recursion overflows while min sample in leaf is mainly for smoothing votes for regression -- the spirit of the method is that

Each tree is grown to the largest extent possible.

  • $\begingroup$ So then, this makes me wonder if Random Forest is even the best way to go about this if it depends on uncorrelated trees? Perhaps an easier separation algorithm like logistic regression or SVM? $\endgroup$ – Hunle Dec 15 '15 at 16:14
  • $\begingroup$ Possibly; yet the only way to be sure is to just make few tests. $\endgroup$ – user88 Dec 15 '15 at 20:11
  • $\begingroup$ How to test that ? $\endgroup$ – Hunle Dec 15 '15 at 21:30
  • $\begingroup$ I find for random forest regression that if OOB-explained variance is lower than 50%, it improves performance slightly to lower bootstrap sample size, and thus reducing also tree depth (and increasing tree decorrelation). A similar but smaller performance improvement can also be achieved limiting 'max depth' or raising 'min node', but I guess the tree decorrelation is not as good. $\endgroup$ – Soren Havelund Welling Dec 17 '15 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.