I am looking at a dataset with ~120 observations and I am investigating it using two sets of explanatory variables, one has about 12 features, the other about 8. This is for a regression analysis.
Setup
Regression analysis on a small dataset, n=120, p=8-12 (non collinear), looking to select the best performing predictive (svm,rf) and inferential (lm, glmnet) algorithims, because I need to explain the data, as well as predict future results.
So far my workflow is
1. Test a few different algorithms, lm, glmnet, rf, svm, (caret) using 10-fold 10-repeat (or so) CV and see which result in the lowest MSE.
2. Find which algorithms perform best and tune the hyperparameters for SVM/rf etc again using a new 10-fold 10-repeat CV (or should I roll this into the previous step?)
3. Report the RMSE/R2 for the final model produced in the previous step as a proxy for performance on unseen data.
Due to my small number of observations, I was hoping to avoid setting aside a test set, although I know it is preferable, and just using 10-fold 10-repeat (or so) Cross validation so I can train on as much of the data as possible.
Is it possible to just use the RMSE estimate from the cross validation allowing me to train on folds with ~108 observations? Or is it still preferable to put aside 10-20% (12-24 observations) for a test set and cross validate on the remaining 96-108 (so around 87-90 per fold) before validating on the holdout/test set. If so, how small can I make the test set and have it still remain valid?
So my questions are
1. Do I need a test set or can I estimate RMSE/R2 etc from Kfold CV.
2. If I do need a test set, how small can I make it? (I will use createDataPartition to ensure it's relatively balanced).
3. Do I need to do a nested CV, and if so what is the best way to do it?
4. Should I tune the hyperparameters at the same time I test the different algorithms? Or do I run it as a seperate set afterwards?
5. Once the models are selected and tuned, I believe I predict on the whole dataset using the tuned model parameters and report this R2/RMSE (and for lm etc the weights of the coefficients), or do I simply report the model performance and coefficient weightings from the final tuned model from CV?