2
$\begingroup$

I've been playing around with the German Credit dataset available in Kuhn & Johnson's caret package for R.

First, I build a very simple model (I'm not using CV to simplify the code; this is taken from their examples in the AppliedPredictiveModeling package):

library(caret)
library(randomForest)
data(GermanCredit)

GermanCredit <- GermanCredit[, -nearZeroVar(GermanCredit)]
GermanCredit$CheckingAccountStatus.lt.0 <- NULL
GermanCredit$SavingsAccountBonds.lt.100 <- NULL
GermanCredit$EmploymentDuration.lt.1 <- NULL
GermanCredit$EmploymentDuration.Unemployed <- NULL
GermanCredit$Personal.Male.Married.Widowed <- NULL
GermanCredit$Property.Unknown <- NULL
GermanCredit$Housing.ForFree <- NULL

set.seed(100)
inTrain <- createDataPartition(GermanCredit$Class, p = .8)[[1]]
GermanCreditTrain <- GermanCredit[ inTrain, ]
GermanCreditTest  <- GermanCredit[-inTrain, ]

set.seed(1056)
credit.rf <- randomForest(Class~., data = GermanCreditTrain, ntree = 500)

Now, if I predict the outcome Class on the test set, and do this several times:

credit.pred1 <- predict(credit.rf, GermanCreditTest)
credit.pred2 <- predict(credit.rf, GermanCreditTest)
credit.pred3 <- predict(credit.rf, GermanCreditTest)

and compare the predictions:

credit.pred1 == credit.pred2
credit.pred2 == credit.pred3
credit.pred1 == credit.pred3

I see that the same model, on the same dataset, makes slightly different decisions.

Why is this? I understand this is normal when building a model using cross-validation (or not, as we unfortunately see all too often), but I thought that when a model was built, it would be making the exact same decisions on a certain test set every time you run it?

EDIT: I tried again as asked, and still have the same problem. Here's the output from sessionInfo():

R version 3.1.1 (2014-07-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252       LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] randomForest_4.6-10 pmml_1.2.30         XML_3.98-1.1        caret_6.0-35        ggplot2_1.0.0       lattice_0.20-29    

loaded via a namespace (and not attached):
 [1] BradleyTerry2_1.0-5 brglm_0.5-9         car_2.0-21          codetools_0.2-8       colorspace_1.2-4    digest_0.6.4       
 [7] foreach_1.4.2       grid_3.1.1          gtable_0.1.2        gtools_3.4.1        iterators_1.0.7     lme4_1.1-7         
[13] MASS_7.3-33         Matrix_1.1-4        minqa_1.2.4         munsell_0.4.2       nlme_3.1-117        nloptr_1.0.4       
[19] nnet_7.3-8          plyr_1.8.1          proto_0.3-10        Rcpp_0.11.3         reshape2_1.4        scales_0.2.4       
[25] splines_3.1.1       stringr_0.6.2       tools_3.1.1      

EDIT 2: This is unbelievable. When I copy the code I posted on CrossValidated for the website, the model produces the same results. But when I copy the code from my code editor (Sublime Text 2.0.2 with the Enhanced-R plugin) into RStudio, I run into the same problem I've described! Here's the address to the Gist I created, it reproduces the problem: https://gist.github.com/anonymous/32b3c8194362d2e10527

$\endgroup$

3 Answers 3

2
$\begingroup$

Random forests use bootstrap sampling to build many different decision trees on the same dataset. While each individual decision would fit the same model to the exact same data, you get a different aggregate model each run because you take different bootstrap samples each run.


Edit: I misunderstood your question. First of all, when I run your code, I DO get the exact same results for credit.pred1, credit.pred2, credit.pred3. Please start a new R session, re-run your code, and check your results. Furthermore, use this code to check equality:

all.equal(credit.pred1, credit.pred2)
all.equal(credit.pred2, credit.pred3)
all.equal(credit.pred1, credit.pred3)

if you STILL get different results, run sessionInfo() and post the results here.

$\endgroup$
0
2
$\begingroup$

I also incurred the same problem randomForest function giving different values for different passes. As Zach mentioned: random forest algorithm randomly creates multiple subsets of the data, the end results might vary slightly for different passes. To overcome this, I simple ran set.seed(500) every time before a new pass, so as to reset the seed to 500 and it is giving me exact mirror results. I hope it helped.

$\endgroup$
1
$\begingroup$

As advised by @John Richardson when I crossposred the question to SO, I tried using the train function from the caret package for building the model:

credit.rf <- randomForest(Class~., data = GermanCreditTrain, ntree = 500)

That solves the problem. But I still have absolutely no idea what's causing it!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.