Which method is better for hyperparameter tuning: sss-fold + independent test set or nested cross-validation?

Question

I have a dataset 21 subjects and 2107 samples. Each subject has a different # of samples, so in order to balance the dataset I did a learning curve. And I found that with 36 samples for the training fold and 4 for the testingfold are enough (I used StratifiedShuffleSplit and my trainig set = 840 samples). So with the leftover epochs I can create a holdout set of 4 epochs for each subject (holdoutset=84 samples).

Now I want to do hyperparameter optimisation and later feature selection. I read this:https://sebastianraschka.com/blog/2016/model-evaluation-selection-part3.html

So which method would be better: 1) Use of sss-fold to make different models by tweaking the hyperparameters, and finally use the holdout set to evaluate the best model?

2) Use nested-cross validation and forget about the holdout set?

Answer 1

Are you training within-subject models or a subject independent model? Regression or classification?

For subject-independent models (i.e. you want to predict unknown subjects): the number of subjects is what you need to consider as independent cases, so you have a total sample size of 21. Most importantly, all splitting needs to be done subject-wise (i.e. split into training subjects and test subjects)
In that case, 21 subjects is probably far too few to do anything in the hold-out line. Actually, I'd expect trouble as well for the hyperparameter optimization because that is affected by the testing variance as well and possibly even more so than the validation of the final model.