Let's say I have 1000 Samples and want to build an ANN. Then I split my dataset into train set (800) and test set (200). After that, I do the following

  1. Cross-validate my train set with different hyperparameter combinations
  2. Pick up the top 3 models and train them using the whole train set
  3. Test the models' accuracy in the test set
  4. The one that perfoms better in the test set is the winner

This is more or less how I saw some people doing. The point is, during step 2, we do not have a way to efficiently tune the weights. I came with the idea of split the initial dataset (1000) into train set (700), validation set (100) and test set (200), and create some additional steps

2.5 Tune the weights of models obtained in 2 using the validation set.

2.75 after the weights are tuned, let the model see the validation set and tweak the models a little bit more.

  1. Test models' accuracy in the test set

I haven't seen anyone doing it, since it seems that cross-validation is made essentially for avoiding the creation of this "extra" validation set splitting, however I cannot figure out how to do weight tuning without this extra validation set.


1 Answer 1


You should train on all your training data at once. Hyperparameter tuning is done by many runs of the training each time on the same dataset, just different parameters. See this explanation: https://www.jeremyjordan.me/hyperparameter-tuning/

  • $\begingroup$ So I should only select the model (nº neurons, AF, weights, etc) that outperformed the others during CV and say "that's the champion" ? $\endgroup$
    – corsetti
    Oct 18, 2020 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.