1
$\begingroup$

I have to compare the two strategies to choose optimal parameters of my NN. I have the following reasoning. Please tell if it is correct!

In train-validation-test split case, the validation dataset is being used to select model hyper-parameters and not to train the model hence reducing the training data might lead to higher bias. Also the validation error deoends a lot on which data points end up in the validation set and hence different validation sets might give different optimal parameters i.e. the evaluation results in higher variance.
In k fold cv , which is a more progressive procedure, each subset and hence every data point is used for validation exactly once. Since the RMSE is averaged over k subsets, the evaluation is less sensitive to the partitioning of data and variance of the resulting estimate is significantly reduced. Also since all the data points are used for training bias is also reduced.

$\endgroup$
  • $\begingroup$ Do you want to compare a single train-validation-test split with nested (aka double) cross validation? $\endgroup$ – cbeleites Apr 2 '18 at 16:33
  • $\begingroup$ No I want to justify to a reviewer why I think k fold cross validation (normal not nested CV) is a better option than single train-validation-test split for validation purpose? I have read some articles on this and prepared this draft. I want to know whether what ive written correct or not. Although it makes sense to me $\endgroup$ – Sarthak Tiwari Apr 2 '18 at 16:57
1
$\begingroup$

While your argumentation why cross validation (or resampling validation in general) is better than a single random split is valid, for the optimization you need nested or double cross validation. That is the analogue to train-valiation-test, i.e. 2 splits producing 3 subsets of your data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.