3
$\begingroup$

To preface this question, I am building a Random Forest model using the randomForest package in R. I am not sure if this question is dependent upon my program, or if it can be answered based on the inherent properties of the algorithm. I felt this question was more appropriate for "Cross Validated" versus "Overflow", please let me know if you think otherwise.

A Random Forest model creates many decision trees which contain a subset of variables and data. Is there any guarantee that every possible combination of variables are accounted for across all of the decision trees?

For example, I have 4 variables that I am feeding into the RF model. Three of those variables are binary (X1, X2, Y1, Y2, Z1, Z2). The other variable (of ordinal type) contains 8 unique values.(A1...A8). This leaves me with 56 possible variable combinations.

How can I guarantee that all possible unique combinations of variables are accounted for across trees? I understand that simply increasing the number of trees would increase the likelihood of this. I also realize that my example contains a relatively small number of variables, however consider a situation with many variables and unique combinations.

Improve this question
$\endgroup$
2
  • $\begingroup$ No, there's no such garanty. The total number of combinations get very big, very fast. $\endgroup$
    – Fernando
    Commented Sep 28, 2017 at 15:01
  • $\begingroup$ @Fernando I would suggest that you expand this comment to an answer, otherwise this question will stay unanswered $\endgroup$
    – psarka
    Commented Sep 28, 2017 at 15:07

2 Answers 2

Reset to default
3
$\begingroup$

No, there's no such garanty. The total number of combinations gets very big, very fast.

The additional questions is: is that even necessary? Given that real data has noise and can present high correlation, forcing all combinations doesn't seem to help much.

If you know (by field experience) that all your variables are important, then maybe a deep neural net could work better.

Improve this answer
$\endgroup$
2
  • $\begingroup$ For my purposes, it would be necessary because I have determined my variables to all be relevant. As you suggest, I will take a look at deep neural networks Thank you for the information. I see how the number of combinations could be very, very large and how it would not be possible to account for all combinations in those situations. $\endgroup$
    – Steve
    Commented Sep 28, 2017 at 15:48
  • $\begingroup$ Deep nets may work well if you have a lot of variables with complex patterns and iterations. $\endgroup$
    – Fernando
    Commented Sep 28, 2017 at 17:33
2
$\begingroup$

A Random Forest model creates many decision trees which contain a subset of variables and data. Is there any guarantee that every possible combination of variables are accounted for across all of the decision trees?

No. Consider the extreme example where your forest consists of a single tree with depth one.

How can I guarantee that all possible unique combinations of variables are accounted for across trees? I understand that simply increasing the number of trees would increase the likelihood of this. I also realize that my example contains a relatively small number of variables, however consider a situation with many variables and unique combinations.

You can't. And, perhaps more importantly, you probably don't want that as well. Doing that would decrease the variance of the models.

Having said that, you could try bagged trees (without attribute bagging, i.e. random selection of features) instead of Random Forests ensuring that all combinations are visited at least once, but I'm not sure this model would perform any better than a full Random Forest.

Improve this answer
$\endgroup$
1
  • $\begingroup$ That makes sense. I was thinking that the process of actually "guaranteeing" each combination was covered would result in a loss in the true randomness of selecting variables/data. I will take a look at bagged trees, thank you for the information. $\endgroup$
    – Steve
    Commented Sep 28, 2017 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.