2
$\begingroup$

I am a bit confused about cross validation. In some papers, cross validation is done after train/test split via splitting the test set into folds. In other papers, cross validation is done without any other splits. If I got the right idea, the first type is to set hyperparameters and the second one is to evaluate model with already selected hyperparameters, is that right?

Let's say I have a CNN model that tends to overfit after some time (100+ epochs) -> I need to monitor val_loss to stop him in right stage to achieve best model. Would in these case be okay to use the first type of cross validation for model selection (split test would be used as validation) or is this considered hacky?

$\endgroup$
2
$\begingroup$

In some papers, cross validation is done after train/test split via splitting the test set into folds. In other papers, cross validation is done without any other splits.

It is likely that you have encountered abuse of terminology as relates to cross validation. The truth is, most neural network papers don't actually do cross validation. In fact I have seen papers at CVPR that do model selection and performance reporting all using a single val/test set. Coming to your question:

Wikipedia article on cross validation has a pretty good summary – "One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, in most methods multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (e.g. averaged) over the rounds to give an estimate of the model's predictive performance."

Given that repeating neural network training multiple times, each time using a different train set drawn from your dataset, can be computationally expensive, the most commonly used model selection strategy is to split the dataset once at the start of the experiment into train/val/test split. Then, you train once on the train set, while tracking performance on the validation set. Model selection is done quite simply, as you said, by picking the model that does best on validation set.

This lecture (~12 mins only) from Andrew Ng describes this very clearly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.