I have a question regarding hold-out vs. cross-validation. I have a dataset with ~650 cases which I am analyzing in R using the caret package. There I have a regression problem and a classification problem.
First I created a training and test dataset with an 80:20 split and then I used cross-validation (10-fold, 3 repeats) on the training dataset to fit the final model. Then I tested the model on the test data set.
The question now is,
- what do I specify as RMSE and AUC ROC?
- The results of the fitted model on the training data set or on the test data set?
I have the feeling that the results in the test data set are very strongly dependent on the coincidence of the split in the training and test data set.
- Also, would it be okay to have only one training dataset?
- If so, how would one plot an AUC ROC and a bland-altman?
I'm a little confused that even though you have cross-validation, you should still have an additional independent test dataset. In principle one has then so to speak a training, validation and test dataset.
- Wouldn't a pure training and validation dataset be possible?