14
$\begingroup$

I am a bit confused: How can the results of a trained Model via caret differ from the model in the original package? I read Whether preprocessing is needed before prediction using FinalModel of RandomForest with caret package? but I do not use any preprocessing here.

I trained different Random Forests by using the caret package and tuning for different mtry values. 

> cvCtrl = trainControl(method = "repeatedcv",number = 10, repeats = 3, classProbs = TRUE, summaryFunction = twoClassSummary)
> newGrid = expand.grid(mtry = c(2,4,8,15))
> classifierRandomForest = train(case_success ~ ., data = train_data, trControl = cvCtrl, method = "rf", metric="ROC", tuneGrid = newGrid)
> curClassifier = classifierRandomForest

I found mtry=15 to be the best parameter on the training_data:

> curClassifier
 ...
Resampling results across tuning parameters:

mtry  ROC    Sens   Spec   ROC SD   Sens SD  Spec SD
 4    0.950  0.768  0.957  0.00413  0.0170   0.00285
 5    0.951  0.778  0.957  0.00364  0.0148   0.00306
 8    0.953  0.792  0.956  0.00395  0.0152   0.00389
10    0.954  0.797  0.955  0.00384  0.0146   0.00369
15    0.956  0.803  0.951  0.00369  0.0155   0.00472

ROC was used to select the optimal model using  the largest value.
The final value used for the model was mtry = 15.

I assessed the model with an ROC Curve and a confusion matrix:

##ROC-Curve
predRoc = predict(curClassifier, test_data, type = "prob")
myroc = pROC::roc(test_data$case_success, as.vector(predRoc[,2]))
plot(myroc, print.thres = "best")

##adjust optimal cut-off threshold for class probabilities
threshold = coords(myroc,x="best",best.method = "closest.topleft")[[1]] #get optimal cutoff threshold
predCut = factor( ifelse(predRoc[, "Yes"] > threshold, "Yes", "No") )


##Confusion Matrix (Accuracy, Spec, Sens etc.)
curConfusionMatrix = confusionMatrix(predCut, test_data$case_success, positive = "Yes")

The resulting Confusion Matrix and Accuracy: 

Confusion Matrix and Statistics
      Reference
Prediction   No  Yes
   No  2757  693
   Yes  375 6684

           Accuracy : 0.8984
 ....

Now I trained a Random Rorest with the same parameters and same training_data using the basic randomForest package:

randomForestManual <- randomForest(case_success ~ ., data=train_data, mtry = 15, ntree=500,keep.forest=TRUE)
curClassifier = randomForestManual

Again I created predictions for the very same test_data as above and assessed the confusion matrix with the same code as above. But now I got different measures:

Confusion Matrix and Statistics

      Reference
Prediction   No  Yes
       No  2702  897
       Yes  430 6480

           Accuracy : 0.8737 
           ....

What is the reason? What am I missing?

|cite|improve this question
$\endgroup$
  • 3
    $\begingroup$ Did you use the same value for the random seed for both models? $\endgroup$ – mmmmmmmmmm Mar 12 '15 at 16:37
  • $\begingroup$ I think so. I set the seed earlier in the code when splitting the data set to training and test data, then trained the caret model, then trained the "original" rf-model. So the seed should stay the same once set in the beginning, shouldn't it? $\endgroup$ – Malte Mar 12 '15 at 17:00
  • $\begingroup$ I tried to insert another set.seed directly before training the "original" rf-model. Doesn't solve the problem, unfortunately. $\endgroup$ – Malte Mar 12 '15 at 18:00
  • 3
    $\begingroup$ You should test this using the seeds argument of trainControl $\endgroup$ – topepo Mar 15 '15 at 20:07
4
$\begingroup$

I think the question while somewhat trivial and "programmatic" at first read touches upon two main issues that very important in modern Statistics:

  1. reproducibility of results and
  2. non-deterministic algorithms.

The reason for the different results is that the two procedure are trained using different random seeds. Random forests uses a random subset from the full-dataset's variables as candidates at each split (that's the mtry argument and relates to the random subspace method) as well as bags (bootstrap aggregates) the original dataset to decrease the variance of the model. These two internal random sampling procedures thought are not deterministic between different runs of the algorithm. The random order which the sampling is done is controlled by the random seeds used. If the same seeds were used, one would get the exact same results in both cases where the randomForest routine is called; both internally in caret::train as well as externally when fitting a random forest manually. I attach a simple code snippet to show-case this. Please note that I use a very small number of trees (argument: ntree) to keep training fast, it should be generally much larger. 

library(caret)

set.seed(321)
trainData <- twoClassSim(5000, linearVars = 3, noiseVars = 9)
testData  <- twoClassSim(5000, linearVars = 3, noiseVars = 9)

set.seed(432)
mySeeds <- sapply(simplify = FALSE, 1:26, function(u) sample(10^4, 3))
cvCtrl = trainControl(method = "repeatedcv", number = 5, repeats = 5, 
                      classProbs = TRUE, summaryFunction = twoClassSummary, 
                      seeds = mySeeds)

fitRFcaret = train(Class ~ ., data = trainData, trControl = cvCtrl, 
                   ntree = 33, method = "rf", metric="ROC")

set.seed( unlist(tail(mySeeds,1))[1])
fitRFmanual <- randomForest(Class ~ ., data=trainData, 
                            mtry = fitRFcaret$bestTune$mtry, ntree=33)

At this point both the caret.train object fitRFcaret as well as the manually defined randomForest object fitRFmanual have been trained using the same data but importantly using the same random seeds when fitting their final model. As such when we will try to predict using these objects and because we do no preprocessing of our data we will get the same exact answers. 

all.equal(current =  as.vector(predict(fitRFcaret, testData)), 
          target = as.vector(predict(fitRFmanual, testData)))
# TRUE

Just to clarify this later point a bit further: predict(xx$finalModel, testData) and predict(xx, testData) will be different if one sets the preProcess option when using train. On the other hand, when using the finalModel directly it is equivalent using the predict function from the model fitted (predict.randomForest here) instead of predict.train; no pre-proessing takes place. Obviously in the scenario outlined in the original question where no pre-processing is done the results will be the same when using the finalModel, the manually fitted randomForest object or the caret.train object. 

all.equal(current =  as.vector(predict(fitRFcaret$finalModel, testData)), 
          target = as.vector(predict(fitRFmanual, testData)))
 # TRUE

all.equal(current =  as.vector(predict(fitRFcaret$finalModel, testData)),
          target = as.vector(predict(fitRFcaret, testData)))
# TRUE

I would strongly suggest that you always set the random seed used by R, MATLAB or any other program used. Otherwise, you cannot check the reproducibility of results (which OK, it might not be the end of the world) nor exclude a bug or external factor affecting the performance of a modelling procedure (which yeah, it kind of sucks). A lot of the leading ML algorithms (eg. gradient boosting, random forests, extreme neural networks) do employ certain internal resampling procedures during their training phases, setting the random seed states prior (or sometimes even within) their training phase can be important.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ The important part was to set the seeds argument explicitly in "trainControl" using the argument "seeds" $\endgroup$ – Malte Jan 2 '17 at 16:28
  • $\begingroup$ Yes, of course. I wanted to make sure that the issue as to why this is needed, was fully clarified. $\endgroup$ – usεr11852 says Reinstate Monic Jan 2 '17 at 17:16
  • $\begingroup$ How do I run train so that it's exactly equivalent to randomForest? I tried method="none" but not sure how to set the seed to the single value. Thanks. $\endgroup$ – Simon Woodward May 15 '19 at 0:15
  • $\begingroup$ Apologies but it is unclear if you have any preProcess or how the randomForest is trained to begin with. In general, assuming we have no pre-processing steps we need to ensure that both the seed and the hyperparameters (here just mtry) used are the same. $\endgroup$ – usεr11852 says Reinstate Monic May 15 '19 at 8:41
0
$\begingroup$

Predictions from curClassifier are not the same as predictions from curClassifier$finalModel link. You have reproduced the finalModel and are comparing it to the predict.train object.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ While what you say is true, it is unfortunately a bit misleading in the current setting as no pre-processing is done by the OP. Both predict should (and do actually) give the same predictions in the case the OP explores. I clarify this point a bit more in my post. $\endgroup$ – usεr11852 says Reinstate Monic Dec 29 '16 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.