Not sure if this post belongs here or if stack overflow would be more appropriate.

I am starting to familiarize with the caret package in R which seems very powerful for the purpose of optimizing and implementing various machine learning methods. According to my understanding the key idea of the package is to train a model across different parameter sets and resampling methods and to select the optimal calibration based on a certain performance measure. This optimized model can subsequently be used to compute predictions on the test data.

Does the package also allow computing predictions for all trained models other than the optimal model?

If this is possible a minimum working example would be nice, but not essential.

The reason for my question is that I am interested in checking the predictive performance of the optimal model relative to the other trained models on the test data. Moreover, I would like to evaluate the performance of forecast combination schemes based on multiple model calibrations.


1 Answer 1


No, in the returned model caret does only provide finalModel as the determined best parametrization trained again on all training data without resampling or similar. Thereby, the final training is the same as if you would have trained this parametrization with trainControl(method='none').

Therefore, what you can do: train those parametrizations you would like to get a test set performance by hand, using trainControl(method='none') and all training data. You could then apply all those models to your test set using predict(model, ...). But keep in mind that you should not compare multiple models based on only the test set performance.

Update: caret provides a good explanation on how to compare multiple models with partitioning + resampling. This could boil down to something like:

training_indexes <- createDataPartition(y = iris$Species, p = 0.8, list = F)
training <- iris[training_indexes,]
testing <- iris[-training_indexes,]
# 2 example models
models <- list()
models$knn <- train(training[,1:4], training[,5], method='knn', tuneGrid=expand.grid(k=1:5), trControl = trainControl(method = 'repeatedcv', 10, 20, savePredictions = T))
models$lda2 <- train(training[,1:4], training[,5], method='lda2', tuneGrid=expand.grid(dimen=1:5), trControl = trainControl(method = 'repeatedcv', 10, 20, savePredictions = T))
# compare models by results of partition+repearts
results <- resamples(x = models)

Models performance

# example of resampling performance of your chosen model in more detail
confusionMatrix(data = models$knn$pred$pred, reference = models$knn$pred$obs)
# your chosen model on test set
confusionMatrix(data = predict(models$knn, newdata = testing[,1:4]), testing[,5])
  • $\begingroup$ Thanks! Is there a good tutorial or paper on how to conduct a proper comparison? If the ultimate goal is to achieve good out-of-sample forecasting performance my intuition tells me that the test set performance should play a crucial role in the forecast evaluation. $\endgroup$
    – kanimbla
    Aug 1, 2016 at 8:00
  • 1
    $\begingroup$ @kanimbla I've added a link to the caret HP in the answer, which focuses on how to evaluate models in caret. This one together with related articles on the caret HP should give you an complete picture of how caret wants us to do things :) The minimal example is mostly here for reference, reading the information on the caret page is still beneficial for sure. $\endgroup$ Aug 1, 2016 at 8:24
  • $\begingroup$ +1 Excellent elaboration! I will have look into the relevant package resources to a get a better idea of how to approach model evaluation in caret. $\endgroup$
    – kanimbla
    Aug 1, 2016 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.