1
$\begingroup$

After following the questions

Below there is a reproducible example using IRIS dataset where I can't reproduce the random forest model when not using caret's train function.

library(doParallel); library(caret)

#create a list of seed, here change the seed for each resampling
set.seed(123)
seeds <- vector(mode = "list", length = 11)#length is = (n_repeats*nresampling)+1
for(i in 1:10) seeds[[i]]<- sample.int(n=1000, 3) #(3 is the number of tuning parameter, mtry for rf, here equal to ncol(iris)-2)

seeds[[11]]<-sample.int(1000, 1)#for the last model

#control list
myControl <- trainControl(method='cv', seeds = seeds, index=createFolds(iris$Species))

#run model in parallel
cl <- makeCluster(12, type="SOCK",outfile="")
registerDoParallel(cl)
model1 <- train(Species~., iris, method='rf', trControl=myControl)

model2 <- train(Species~., iris, method='rf', trControl=myControl)
stopCluster(cl)

#compare
all.equal(predict(model1, type='prob'), predict(model2, type='prob'))

...

 >[1] TRUE

...

# using the same seed as for model 2
set.seed(seeds[[11]])
model3 <- randomForest(Species~., iris, mtry = model2$bestTune$mtry)
all.equal(predict(model2, type='prob'),as.data.frame(predict(model3, type='prob')))

...

>[1] "Component “versicolor”: Mean relative difference: 0.3006435"
>[2] "Component “virginica”: Mean relative difference: 0.2400822" 
  • What am I missing ? Is it possible to reproduce the random forest model returned by caret's train function?
  • If not, can we train the final model in caret with different parameters ? such as importance and proximities flags to true ?
$\endgroup$

closed as off-topic by COOLSerdash, Greenparker, whuber Jul 18 '16 at 17:57

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – COOLSerdash, Greenparker, whuber
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Removing the parallel (i.e. every line containing cl) makes the result reproducible $\endgroup$ – RUser4512 Dec 30 '15 at 15:55
  • $\begingroup$ Sorry, but it is no so. Making it sequential does not make it reproducible. $\endgroup$ – sc25 Dec 30 '15 at 19:56
  • $\begingroup$ Strange, it worked on my machine... I will give it another try on another computer... $\endgroup$ – RUser4512 Dec 30 '15 at 23:07
0
$\begingroup$

I've figured it out in the meantime.

So calling

predict(model1, type='prob')

or model2 for that matter, will evaluate the whole training data. It is the same as

predict(model2,type='prob') == predict(model2$finalModel, iris,type='prob')

Using the predict function on the random Forest model will evaluate the OOB data, so

predict(model2$finalModel, type='prob') == as.data.frame(predict(model3, type='prob'))

Which leads to

predict(model2$finalModel, iris,type='prob') == predict(model3,type='prob')

and this was what was missing when comparing the models results and assessing for reproducibility.

I've left out the call to all.equal in the code above for clarity.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.