My understanding of RF is you take a random selection of the data, as well as a random selection of the feature set. You then take the average of these decision trees to get a good guess, and your overfit should be kept in check by the fact that the trees are looking at different data/features.

But why is it necessary to do this randomly? Couldn't you just as well calculate all the permutations (up to some limit) and run the model on that?


To make it a bit more concrete, let's say I have some dataset with a billion rows of data and 10 features.

I could choose something like 2 features randomly selected. If I do that, there are 45 (10c2) pairs of features a tree could be picking. Now knowing there's 45 pairs, why don't I just evenly sample each pair, instead of sampling randomly? I'm not going to get a completely even set if I do say a random 4500 trees. But I could just say I'll do 100 trees in each feature pair.

  • 2
    $\begingroup$ If you have a large enough set of data, you will hit your "up to some limit" rather quickly $\endgroup$ – Henry Jan 1 '16 at 21:57
  • $\begingroup$ Sure, and if some samples give a vastly different result than others maybe that's saying something about your model. But what about the feature set? If you have 10 features, you could easily find all the models using exactly 2 or 3 or something like that? $\endgroup$ – Carlos Jan 1 '16 at 21:59
  • $\begingroup$ I suppose the answer is that if you aren't overfitting, you're unlikely to get any random sampling that differs much from any other? $\endgroup$ – Carlos Jan 1 '16 at 22:03
  • 14
    $\begingroup$ Even surprisingly modest sets of data (in terms of n and p) yield astronomically large numbers of possible subsamples $\times$ feature-sets. Dealing with all such permutations is only going to be possible for quite small problems. $\endgroup$ – Glen_b Jan 1 '16 at 22:08
  • 1
    $\begingroup$ Look at the number of permutations. nCk explodes very quickly, and you only have a finite amount of RAM to work with. $\endgroup$ – RegressForward Jan 2 '16 at 4:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.