Do I need a validation set if I am doing 10-fold cross validation?

Question

I am looking at a dataset with ~120 observations and I am investigating it using two sets of explanatory variables, one has about 12 features, the other about 8. This is for a regression analysis.

Setup
Regression analysis on a small dataset, n=120, p=8-12 (non collinear), looking to select the best performing predictive (svm,rf) and inferential (lm, glmnet) algorithims, because I need to explain the data, as well as predict future results.

So far my workflow is
1. Test a few different algorithms, lm, glmnet, rf, svm, (caret) using 10-fold 10-repeat (or so) CV and see which result in the lowest MSE.
2. Find which algorithms perform best and tune the hyperparameters for SVM/rf etc again using a new 10-fold 10-repeat CV (or should I roll this into the previous step?)
3. Report the RMSE/R2 for the final model produced in the previous step as a proxy for performance on unseen data.

Due to my small number of observations, I was hoping to avoid setting aside a test set, although I know it is preferable, and just using 10-fold 10-repeat (or so) Cross validation so I can train on as much of the data as possible.

Is it possible to just use the RMSE estimate from the cross validation allowing me to train on folds with ~108 observations? Or is it still preferable to put aside 10-20% (12-24 observations) for a test set and cross validate on the remaining 96-108 (so around 87-90 per fold) before validating on the holdout/test set. If so, how small can I make the test set and have it still remain valid?

So my questions are
1. Do I need a test set or can I estimate RMSE/R2 etc from Kfold CV.
2. If I do need a test set, how small can I make it? (I will use createDataPartition to ensure it's relatively balanced).
3. Do I need to do a nested CV, and if so what is the best way to do it?
4. Should I tune the hyperparameters at the same time I test the different algorithms? Or do I run it as a seperate set afterwards?
5. Once the models are selected and tuned, I believe I predict on the whole dataset using the tuned model parameters and report this R2/RMSE (and for lm etc the weights of the coefficients), or do I simply report the model performance and coefficient weightings from the final tuned model from CV?

Answer 1

1) and 2) Usually it's better to have a test set because it is common to overfit on the validation set (since the hyperparameters are indeed chosen to minimize the validation loss). In your case, however, I would not bother with a test set but it then becomes important not to try too many sets of hyperparameters (that's when overfitting on the validation set usually happens and you get overoptimistic loss).

3) I think nested cross val is essentially using a test set. As I mentioned, it could be just fine with a regular CV.

4) Not sure what you mean here.

5) Here I believe you are confusing a concept. You should indeed "simply report the model performance and coefficient weightings from the final tuned model from CV". The case where you would want to re-train on the whole thing (training + val) is when you have a test set and you make your final predictions on the test set. This is commonly done when you have little data but I think in deep learning these days, they just use the trained model only on the training data rather than re-training it on the training+validation if that makes sense.

Answer 2

There's a (rather common) confusion of what "having a test set" means, due to having 2 separate considerations here:

1. "test set" can mean that generalization error is estimated via some kind of data that is (more-or-less) independent of the data used for training.
   I.e., test set as opposed to autoprediction error.
2. "test set" (aka hold out) can also refer to setting aside such a test set (1) by reserving a single such test set as opposed to generating a number of test (sub)sets in resampling validation (cross valiation, out-of-bootstrap),
   i.e. test set as opposed to cross validation.

Another thing you need to keep in mind is that more-or-less randomly setting aside part of your training data often doesn't lead to as independent a split as we'd like to think. A single split of your available data is far more similar to cross validation than to obtaining a truly independent data set for validation purposes after the training is finished. In my experience, the single split will not avoid any of the mistakes in splitting that lead to data leakage (= dependence) that you'd make in cross validation. From that point of view, what stays are the advantages of cross validation in terms of larger number of tested cases and the possibility to check stability.

---

 1. Do I need a test set or can I estimate RMSE/R2 etc from Kfold CV.

What you need is a test set 1., but you can use cross validation to generate it (test set 2. is not needed).

 2. If I do need a test set, how small can I make it? (I will use createDataPartition to ensure it's relatively balanced).

The size of the tested data (for all test sets 1. So, including cross validation - although there test set size = training set size after a full run) determines the precision/uncertainty of your error estimate.
For RMSE and R² (you are aware of the very limited amount of information you can get from R², are you?) you'll have to measure that uncertainty from the validation results. If the uncertainty on your RMSE is unacceptably high, you'll need more test cases. (For some classification figures of merit such as accuracy or senstivity, specificity, etc. back-of-the-envelope calculations can be done beforehand)

 3. Do I need to do a nested CV,

You don't need it in the sense that any other test set 1. that is independent of the data-driven model optimization would avoid the optimistic bias of "recycling" the error estimates you already used as basis for that optimization.

However, cross validation is typically one of the best strategies if your sample base is limited.

what is the best way to do it?

IMHO the best way to implement the nested cross valiation is to implement

1. a function that wraps your training in the narrower sense together with the inner cross validation into a high-level training function, and then

2. a normal cross validation to cross-validate that high-level training function.

   • this gives you a correct implementation of the nested cross validation, and also
   • avoids confusion as to what to do in terms of your question 5.

4. Should I tune the hyperparameters at the same time I test the different algorithms? Or do I run it as a seperate set afterwards?

Do that together:

• you'll need to optimize the hyperparameters in order to pick the best algorithm - otherwise you'do a rather nonsensical selection between models based on different algorithms of which you don't know how much worse they are than what that algorihm can do with your data, and
  • doing this again in separate levels (again with train/test splits) would result in an unnecessary further reduction of your training base.

Also keep in mind: the uncertainty of the (inner) figures of merit determines what kind of decisions you can reliably do in your optimization. With a total of 120 cases, I'd not be surprised at all if you find that once the hyperparameters are tuned for each of your algorithms at least a bunch of them do similarly well.

5. Once the models are selected and tuned, I believe I predict on the whole dataset using the tuned model parameters and report this R2/RMSE (and for lm etc the weights of the coefficients), or do I simply report the model performance and coefficient weightings from the final tuned model from CV?

I'd recommend to run your high-level training wrapper on the whole data set. Then check whether those tuning results are in line with the tuning results you got in the cross validation.