I am using random forest on high-dimensional grouped data (50 numeric input variables) which have a hierachical structure. The data were collected with 6 replications at 30 positions of 70 different objects resulting in 12600 data points, which are not independent.

It seems random forest is over-fitting the data, since the oob error is much smaller than the error which we get when leaving data from one object out during training and then predicting the outcome of the left out object with the trained random forest. Moreover I have correlated residuals.

I think the overfitting is caused since random forest is expecting independent data. Is it possible to tell the random forest about the hierarchical structure of the data? Or is there another powerful ensemble or shrinkage method that can handle high-dimensional grouped data with a strong interaction structure?

Any hint how I can do better?

  • $\begingroup$ What's the nature of the hierarchical data? Does it allow you to use the leaves of the data as your data points? $\endgroup$
    – casperOne
    Dec 18 '11 at 16:26
  • 1
    $\begingroup$ Have you considered bootstrapping the highest level of the hierarchy, rather than the individual? $\endgroup$ Jun 21 '17 at 1:30

Very late to the party as well, but I think that could be related to something I did a few years ago. That work got published here:


and is about dealing with variable correlation into ensemble of decision trees. You should have a look at the bibliography which is pointing to many proposal to deal with this type of issues (which is common in the "genetic" area).

The source code is available here (but is not really maintained anymore).


Over-Fitting of the Random Forest can be caused by different reasons, and it highly depends on the RF parameters. It is not clear from your post how you tuned your RF.

Here are some tips that may help:

  1. Increase the number of trees

  2. Tune the Maximum Depth of the trees. This parameter highly depends on the problem at hand. Using smaller trees can help with overfitting problem.

  • 2
    $\begingroup$ Very late to the party, but this answer will not solve any problems due to a hierarchical nature of the data set. $\endgroup$ May 17 '14 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.