2
$\begingroup$

Hi I am fairly new to Random Forest estimation, but I could not find a questions similiar to mine. I was surprised that the predictions are different using the same predictors. I would have expected the same. I understand, that the model would be different with each estimation, but getting different predictions for the same predictors?

library(randomForest)
set.seed(100)
df<-mtcars
rt.est<-randomForest(mpg ~ ., 
                     data = df,
                     ntree=1000)
predict(rt.est)
df.double<-rbind(df,df[32,])
rt.est<-randomForest(mpg ~ ., 
                     data = df.double,
                    ntree=1000)
predict(rt.est)    

The results for the last Observation on the Volvo142E are similiar but not the same. Why?

$\endgroup$
  • $\begingroup$ Once you have an answer to the question as stated, you shouldn't really change the substance of the question in a way that breaks the connection with the existing answers. If you wan to ask another question as a fork of this, just start a new thread. You can link back for context, if you want. $\endgroup$ – gung Apr 26 at 19:42
  • $\begingroup$ I did not alter the questions substantially. It was always about having a different prediction for the same predictors. $\endgroup$ – Max M Apr 26 at 20:05
5
$\begingroup$

If you supply the argument newdata the discrepancy disappears: predict(rt.est, newdata=df) gives

Volvo 142E 
22.15557 

Volvo 142E1 
22.15557

When you do not supply newdata, it's probably reporting the out-of-bag results, but I haven't found an explicit clarification of this in the documentation. Samples that are "in-bag" were included in a tree during training as a result of the bootstrap re-sampling procedure; out-of-bag samples were omitted.

We can verify that this is out-of-bag data by calling rt.est$predicted which reports the out-of-bag predictions. The results match predict(rt.est).

Volvo 142E 
22.83609 

Volvo 142E1 
22.85975 

One way to think of predict.randomForest is that it's a shortcut to the out-of-bag predictions unless you supply newdata.


OP originally asked about 2 different ensembles of random forests. This portion of the answer addresses why 2 random forest ensembles might make different predictions on the same data.

The trees are different.

First, randomForest is a random procedure: both the samples chosen for each tree are different (bootstrap resampling), and the features chosen at each split are chosen at random (randomized feature subspaces). Without fixing the random seed, we would expect two randomForest runs to produce different results with high probability for the same reason that flipping a fair coin 1000 times will plausibly result in a different sequence of heads and tails. (You have fixed the seed, however the two instances of randomForest will still be different because the random state is altered after the first randomForest is produced.)

Second, The data used to train the models is different. It appears that you've added an additional row. Different data makes for a different model, which makes for a different prediction. When randomForest conducts its bootstrapping, the probability that df[32,] is in-bag for that tree is larger than for each non-duplicated sample. This change to the data will also change the trees, because choices about where to make splits will be influenced by the increased prominence of this sample.

Different trees make different predictions.

Having the same feature values is only half the battle. The other half is how the trees are constructed.

As an example, suppose I have 3 trees constructed with the random forest procedure, each with 1 split.

  1. This tree has a bootstrap resample and randomly samples cyl, disp and hp. It picks splitting on cyl at 5 as the best split.
  2. This tree has a bootstrap resample and randomly samples cyl, disp and hp. It picks splitting on cyl at 7 as the best split.
  3. This tree has a bootstrap resample and randomly samples wt, disp and hp. It picks splitting on hp at 123 as the best split.

Clearly there will be different predictions whenever the splits change the decision of a sample. A sample with cyl 6 might go "right" for tree 1, but "left" for tree 2. Feature hp doesn't have a one-to-one relationship to cyl, so a split on hp won't generally match splits on cyl.

$\endgroup$
  • $\begingroup$ That i understand. I added one Observation ran the model again and would have expected the same result for the Observation #32 because it has the same predictors. $\endgroup$ – Max M Apr 26 at 14:19
  • $\begingroup$ If you have two completely different decision trees, are they guaranteed to make the same predictions? $\endgroup$ – Sycorax Apr 26 at 14:41
  • $\begingroup$ I edited my Code to make it clearer what confuses me. I do understand that each run of the randomforest command yields a different model if I do not set a seed, but why is the prediction for my last two observations different? So you are saying that the prediction can slighly deviate at the cutoff Point? But this would be more likely for categorical predictors then? $\endgroup$ – Max M Apr 26 at 18:42
  • $\begingroup$ Oh thats surprising to me. So ist a using kind of a mixed random drawn sample to get the predictions for the model? Is this not contradicting your original answer then??? $\endgroup$ – Max M Apr 26 at 18:54
  • $\begingroup$ No, that's absolutely not what it's doing. Please read the explanation in my answer. The behavior of the function changes depending on what arguments you do or do not give to it. Your original question asked about a comparison between 2 random forest ensembles. Your revised question asks about how the function predict works. $\endgroup$ – Sycorax Apr 26 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.