16
$\begingroup$

I am using random forest on high-dimensional grouped data (50 numeric input variables) which have a hierachical structure. The data were collected with 6 replications at 30 positions of 70 different objects resulting in 12600 data points, which are not independent.

It seems random forest is over-fitting the data, since the oob error is much smaller than the error which we get when leaving data from one object out during training and then predicting the outcome of the left out object with the trained random forest. Moreover I have correlated residuals.

I think the overfitting is caused since random forest is expecting independent data. Is it possible to tell the random forest about the hierarchical structure of the data? Or is there another powerful ensemble or shrinkage method that can handle high-dimensional grouped data with a strong interaction structure?

Any hint how I can do better?

Improve this question
$\endgroup$
2
  • $\begingroup$ What's the nature of the hierarchical data? Does it allow you to use the leaves of the data as your data points? $\endgroup$
    – casperOne
    Commented Dec 18, 2011 at 16:26
  • 1
    $\begingroup$ Have you considered bootstrapping the highest level of the hierarchy, rather than the individual? $\endgroup$
    – generic_user
    Commented Jun 21, 2017 at 1:30

2 Answers 2

Reset to default
1
$\begingroup$

Very late to the party as well, but I think that could be related to something I did a few years ago. That work got published here:

http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0093379

and is about dealing with variable correlation into ensemble of decision trees. You should have a look at the bibliography which is pointing to many proposal to deal with this type of issues (which is common in the "genetic" area).

The source code is available here (but is not really maintained anymore).

Improve this answer
$\endgroup$
-1
$\begingroup$

Over-Fitting of the Random Forest can be caused by different reasons, and it highly depends on the RF parameters. It is not clear from your post how you tuned your RF.

Here are some tips that may help:

  1. Increase the number of trees

  2. Tune the Maximum Depth of the trees. This parameter highly depends on the problem at hand. Using smaller trees can help with overfitting problem.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Very late to the party, but this answer will not solve any problems due to a hierarchical nature of the data set. $\endgroup$
    – cbeleites
    Commented May 17, 2014 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.