2
$\begingroup$

I am sorry if there is an obvious or intuitive answer to this, which I missed.

We have tuned the hyperparameters of a RF using Grouped 10 - Fold CV (repeated 5 times), to obtain the values for mtry and ntree which result in the highest accuracy.

I understand the concept of (grouped) K-Fold Cross Validation. However my first basic question is how is the hyperparameter tuning from caret incorporated into this resampling technique? In a toy example where we want to test mtry = c(1,2) and ntree = c(500,1000) (meaning 4 possible combinations of hyperparameters) would caret run the complete 10 Fold CV repeated 5 times, for each of the 4 possible combination of mtry and ntree?

Or does caret iterate through the different hyperparameter combinations in the repeats of the CV?

Furthermore if we want to make predictions about a novel dataset (df2) based on the rf model (output from caret hyperparam tuning), which model will be used? Since in a 10-Fold CV there are 10 models trained on (10-1 Fold of data) and evaluated in 1 fold. I know that the performance metrics are averaged, however I can't really imagine that the same is true for the model.

transfer <- predict(rf.df1,newdata = df2)
confusionMatrix(df2$Class,transfer)
#which model is used here for the predictions?
$\endgroup$

1 Answer 1

1
$\begingroup$

The model used is the one trained on the whole of the training sample using the hyperpameters yielding the optimal average performance based on the CV procedure defined.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.