Skip to main content
edited title
Link
Björn
  • 425
  • 2
  • 16

Predictions based on CVk-fold Cross Validation, which model is used (Caret)

Source Link
Björn
  • 425
  • 2
  • 16

Predictions based on CV which model is used (Caret)

I am sorry if there is an obvious or intuitive answer to this, which I missed.

We have tuned the hyperparameters of a RF using Grouped 10 - Fold CV (repeated 5 times), to obtain the values for mtry and ntree which result in the highest accuracy.

I understand the concept of (grouped) K-Fold Cross Validation. However my first basic question is how is the hyperparameter tuning from caret incorporated into this resampling technique? In a toy example where we want to test mtry = c(1,2) and ntree = c(500,1000) (meaning 4 possible combinations of hyperparameters) would caret run the complete 10 Fold CV repeated 5 times, for each of the 4 possible combination of mtry and ntree?

Or does caret iterate through the different hyperparameter combinations in the repeats of the CV?

Furthermore if we want to make predictions about a novel dataset (df2) based on the rf model (output from caret hyperparam tuning), which model will be used? Since in a 10-Fold CV there are 10 models trained on (10-1 Fold of data) and evaluated in 1 fold. I know that the performance metrics are averaged, however I can't really imagine that the same is true for the model.

transfer <- predict(rf.df1,newdata = df2)
confusionMatrix(df2$Class,transfer)
#which model is used here for the predictions?