Caret package - Is it possible to compute predictions for non-optimal models?

Question

Not sure if this post belongs here or if stack overflow would be more appropriate.

I am starting to familiarize with the caret package in R which seems very powerful for the purpose of optimizing and implementing various machine learning methods. According to my understanding the key idea of the package is to train a model across different parameter sets and resampling methods and to select the optimal calibration based on a certain performance measure. This optimized model can subsequently be used to compute predictions on the test data.

Does the package also allow computing predictions for all trained models other than the optimal model?

If this is possible a minimum working example would be nice, but not essential.

The reason for my question is that I am interested in checking the predictive performance of the optimal model relative to the other trained models on the test data. Moreover, I would like to evaluate the performance of forecast combination schemes based on multiple model calibrations.

Answer 1

No, in the returned model caret does only provide finalModel as the determined best parametrization trained again on all training data without resampling or similar. Thereby, the final training is the same as if you would have trained this parametrization with trainControl(method='none').

Therefore, what you can do: train those parametrizations you would like to get a test set performance by hand, using trainControl(method='none') and all training data. You could then apply all those models to your test set using predict(model, ...). But keep in mind that you should not compare multiple models based on only the test set performance.

Update: caret provides a good explanation on how to compare multiple models with partitioning + resampling. This could boil down to something like:

library(caret)
set.seed(123456)
training_indexes <- createDataPartition(y = iris$Species, p = 0.8, list = F)
training <- iris[training_indexes,]
testing <- iris[-training_indexes,]
# 2 example models
models <- list()
models$knn <- train(training[,1:4], training[,5], method='knn', tuneGrid=expand.grid(k=1:5), trControl = trainControl(method = 'repeatedcv', 10, 20, savePredictions = T))
models$lda2 <- train(training[,1:4], training[,5], method='lda2', tuneGrid=expand.grid(dimen=1:5), trControl = trainControl(method = 'repeatedcv', 10, 20, savePredictions = T))
# compare models by results of partition+repearts
results <- resamples(x = models)
bwplot(results)

# example of resampling performance of your chosen model in more detail
confusionMatrix(data = models$knn$pred$pred, reference = models$knn$pred$obs)
# your chosen model on test set
confusionMatrix(data = predict(models$knn, newdata = testing[,1:4]), testing[,5])