The approach that I learned (from this Coursera course) is that you divide your dataset into 3 subsets: training, testing, and validation. I think a 60-20-20 ratio between the datasets is common. The two central rules are (a) never use the test set to directly fit a model, and (b) only touch the validation set once, at the very end of the process and after the final model has been chosen.
The training set is used to fit models. This is done using cross-validation:
- Select a range of hyperparameter values and a target statistic.
- Fit the model using CV at each hyperparameter value, optimizing the target statistic. Select the model (hyperparameter value) that optimizes the target statistic.
caret::train
is designed to conduct steps 1 and 2. Use the metric
argument to identify the target statistic (and maximize
to tell the CV algorithm whether to maximize or mining it). Use tuneGrid
and tuneLength
to set the range of hyperparameter values to search over. trControl
lets use fine-tune the process of using CV to select the optimal model. I don't think ROC is one of the pre-defined options for metric
, so you'll need to pass the trControl
argument a trainControl
object, with the summaryFunction
argument of trainControl
a ROC function.
The test set is used iteratively with the training set to refine the model.
- Use the selected model from step 2 to generate predictions on the test set. If the target statistic comes out too low, repeat steps 1 and 2 to fine-tune your model. If the target statistic is sufficiently high, you have your final model.
caret
has three prediction functions to use with this step: extractPrediction
, extractProb
, and predict
. Functions such as plotObsVsPred
and confusionMatrix
can be used to identify problem cases.
This iterative process can introduce overfitting, which can bias your estimates of how your model performs on totally new data. The test set isn't used directly to fit the model, but the iterative process means that you are selecting the model that best fits the test set. The validation set helps avoid that problem.
- Only do this step once, at the very end of your analysis. Use the final model from step 3 to generate predictions on the validation set. Report the results as your out-of-sample accuracy/error estimates.
AFAIK caret
doesn't have specific functions for this final step. But the same functions used with step 3 are useful here.