I have a dataset 21 subjects and 2107 samples. Each subject has a different # of samples, so in order to balance the dataset I did a learning curve. And I found that with 36 samples for the training fold and 4 for the testingfold are enough (I used StratifiedShuffleSplit and my trainig set = 840 samples). So with the leftover epochs I can create a holdout set of 4 epochs for each subject (holdoutset=84 samples).
Now I want to do hyperparameter optimisation and later feature selection. I read this:https://sebastianraschka.com/blog/2016/model-evaluation-selection-part3.html
So which method would be better: 1) Use of sss-fold to make different models by tweaking the hyperparameters, and finally use the holdout set to evaluate the best model?
2) Use nested-cross validation and forget about the holdout set?