Let's say I have 1000 Samples and want to build an ANN. Then I split my dataset into train set (800) and test set (200). After that, I do the following
- Cross-validate my train set with different hyperparameter combinations
- Pick up the top 3 models and train them using the whole train set
- Test the models' accuracy in the test set
- The one that perfoms better in the test set is the winner
This is more or less how I saw some people doing. The point is, during step 2, we do not have a way to efficiently tune the weights. I came with the idea of split the initial dataset (1000) into train set (700), validation set (100) and test set (200), and create some additional steps
2.5 Tune the weights of models obtained in 2 using the validation set.
2.75 after the weights are tuned, let the model see the validation set and tweak the models a little bit more.
- Test models' accuracy in the test set
I haven't seen anyone doing it, since it seems that cross-validation is made essentially for avoiding the creation of this "extra" validation set splitting, however I cannot figure out how to do weight tuning without this extra validation set.