I am a bit confused: How can the results of a trained Model via caret differ from the model in the original package? I read Whether preprocessing is needed before prediction using FinalModel of RandomForest with caret package? but I do not use any preprocessing here.
I trained different Random Forests by using the caret package and tuning for different mtry values.
> cvCtrl = trainControl(method = "repeatedcv",number = 10, repeats = 3, classProbs = TRUE, summaryFunction = twoClassSummary)
> newGrid = expand.grid(mtry = c(2,4,8,15))
> classifierRandomForest = train(case_success ~ ., data = train_data, trControl = cvCtrl, method = "rf", metric="ROC", tuneGrid = newGrid)
> curClassifier = classifierRandomForest
I found mtry=15 to be the best parameter on the training_data:
> curClassifier
...
Resampling results across tuning parameters:
mtry ROC Sens Spec ROC SD Sens SD Spec SD
4 0.950 0.768 0.957 0.00413 0.0170 0.00285
5 0.951 0.778 0.957 0.00364 0.0148 0.00306
8 0.953 0.792 0.956 0.00395 0.0152 0.00389
10 0.954 0.797 0.955 0.00384 0.0146 0.00369
15 0.956 0.803 0.951 0.00369 0.0155 0.00472
ROC was used to select the optimal model using the largest value.
The final value used for the model was mtry = 15.
I assessed the model with an ROC Curve and a confusion matrix:
##ROC-Curve
predRoc = predict(curClassifier, test_data, type = "prob")
myroc = pROC::roc(test_data$case_success, as.vector(predRoc[,2]))
plot(myroc, print.thres = "best")
##adjust optimal cut-off threshold for class probabilities
threshold = coords(myroc,x="best",best.method = "closest.topleft")[[1]] #get optimal cutoff threshold
predCut = factor( ifelse(predRoc[, "Yes"] > threshold, "Yes", "No") )
##Confusion Matrix (Accuracy, Spec, Sens etc.)
curConfusionMatrix = confusionMatrix(predCut, test_data$case_success, positive = "Yes")
The resulting Confusion Matrix and Accuracy:
Confusion Matrix and Statistics
Reference
Prediction No Yes
No 2757 693
Yes 375 6684
Accuracy : 0.8984
....
Now I trained a Random Rorest with the same parameters and same training_data using the basic randomForest package:
randomForestManual <- randomForest(case_success ~ ., data=train_data, mtry = 15, ntree=500,keep.forest=TRUE)
curClassifier = randomForestManual
Again I created predictions for the very same test_data as above and assessed the confusion matrix with the same code as above. But now I got different measures:
Confusion Matrix and Statistics
Reference
Prediction No Yes
No 2702 897
Yes 430 6480
Accuracy : 0.8737
....
What is the reason? What am I missing?
seeds
argument oftrainControl
$\endgroup$