0
$\begingroup$

I have came across a work that did something in their cross validation protocol which I found strange: they run 3-fold cross-validation, but for each fold they used different hyperparameters during training. Specifically, the changed hyperparameters (to train their neural network) were:

  • Number of training epochs.
  • Use of early stopping - for one of the folds they pick the epoch with best validation performance among 300 epochs, while the other two folds run for 150 and 500 epochs respectively, and they just pick the model of the last epoch.

I don't recall ever seeing this sort of thing for cross-validation protocols, so I am wondering if this kind of protocol is outright wrong or just weird.

$\endgroup$

3 Answers 3

2
$\begingroup$

I also react by thinking this is odd. You normally want to test a hyperparameter such as number of epochs across all folds and cross-validation then gives you an estimate of how such a training procedure with these hyperparameters tends to perform. If you still need to pick the hyperparameter value, then you'd pick the one that performs best across folds, which does not seem to be what is going on in the case you describe. That's unless there is some rule they use (and then the specifics of the rule are the hyperparameter(s)), where based on the training data the number of epochs gets chosen (you don't seem to mention anything like that). To avoid leakage of validation performance, such a rule shouldn't be early stopping based on validation set performance (unless splitting within the training part of a cross-validation split is done).

If there's no such rule and the hyperparameters are just "randomly" different across folds, I have no idea what the cross-validation actually estimates (without such a rule, it would be a wrong estimate of the performance of the model trained in a way that one could ever apply in practice) and it would to me raise a concern that we might just be seeing a result of "cross-validation-performance-hacking". Of course, without details and the authors' explanation why they did things the way they did (perhaps there's some kind of rationale that might somehow make sense, once the authors disclose it???), it's hard to know.

$\endgroup$
1
  • $\begingroup$ After more analysis, there are two issues with the author's protocol: first, is the fact that they name it cross-validation, but it isn't really cross-validation but rather just repeating a 60/20/20 split multiple times. In this scenario, they proceed in the way that I described for each of the splits, using different hyperparameters during triaining, which I would say is the second issue/weirdness, and then aggregate the performance on the test set. Not exactly wrong, just an odd protocol to use. There is no reasoning in the text for usage of this kind of protocol. $\endgroup$
    – Alb
    Commented Jul 24, 2023 at 4:31
1
$\begingroup$

It's weird, and IMHO one should not call it cross validation since cross validation implies that the same is done to all folds.

It is also a rather inefficient heuristic, since candidate models are compared with test data based on only 1/3 of the available training + optimization data. Also, the training sets across those different hyperparameter choices are neither the same nor independent. Both makes statistical comparisons difficult and inefficient.

But IMHO: during training and optimization one can pretty much do whatever one likes - as long as there is a honest test (proper validation with statistically independent cases) of the final model's generalization performance - that's where one pays for bad training/optimization decisions...

So, I'd reserve "outright wrong" here for the situation where the final model is not properly tested.

$\endgroup$
1
  • $\begingroup$ That was honestly my feeling too. Not inherently wrong (since they did have a proper hold-out test set), just weird and inneficient. Taking a closer look, I noticed that the main issue is that the work uses the term cross-validation wrong. They aren't really doing cross-validation, but rather just repeats of random 60/20/20 splits, training on each split (each split having the hyperparameter difference I described), and then aggregating the answer on the test set by averaging. Overall, not a protocol I'd follow or have seen followed before, but not inherently wrong. $\endgroup$
    – Alb
    Commented Jul 24, 2023 at 4:06
0
$\begingroup$

The question of whether it is outright wrong is difficult. In the case that the data in all folds is very similar, nothing is wrong. However, there is completely no need to make that assumption.

The goal of cross-validation is to experiment with which hyperparameters the model is performing the best. In order to have "fair competition" between all combinations of hyperparameters, you would want them to have the same training datasets and the same validation dataset. Otherwise, one set of hyperparameters might have an "easier" validation set.

So my advice: split the dataset into 2/3 training and 1/3 validation to experiment with different hyperparameters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.