0
$\begingroup$

I have a paper coming up and I would like to clear some questions regarding cross-validation, because I could not find this information explicitly stated anywhere in literature or it differs.

  1. I understand that cross-validation is used for finding the optimal hyperparameters of a model*. Does this mean that for every training/validation split of the cross-validation, a new model (with reinitialized weights) with the current set of hyperparameters to be evaluated, is trained? For example, using a 10-fold cross-validation, we have trained 10 different models with 10 different training/validation splits but with the same hyperparameters. This brings me to the second question.

  2. After we have trained 10 models and achieved 10 performance scores in a cross-validation, do we average the scores, or take the model with the best score and evaluate it against a test set that was withheld in the beginning?

  3. Since I would like to test my model (lets say I have found the hyperparameters) on a known dataset, and this dataset doesn't have evaluation benchmarks (which samples to take for training and which for test), is a valid evaluation to do a 10-fold cross-validation on this dataset, without withholding a test set in the beginning, and plot a confusion matrix for each training/validation split, summing all matrices in the end element-wise? An alternative evaluation would be to average all 10 accuracies returned.

*the model is of a convolutional neural network, the task is classification

$\endgroup$
0
$\begingroup$

My general advice would be to transparent in how you report your findings. There are standards in different scientific communities, but there is no single way of performing cross-validation. That being said, let me try my best to answer your three questions:

  1. Generally, when using cross-validation, you need to split in to THREE groups. Training, validation, and test. The issue is that generally when training a CNN, you stop training after the model stopes improving when evaluated against the validation data. However, if the validation data is not a fully representative sample of the problem space, then your evaluation metrics will appear better than they are. So, when performing 10-fold cross validation, train on 8 folds, validate on 1, and test on 1. Your metrics against the test fold is what you should be reporting.

  2. This is an interesting idea, however, it isn't really cross-validation. Not that there is something intrinsically wrong with this methods, but assuming that your train/test/validation split is sufficiently random, your ten models should be relatively similar. This method of evaluation also isn't a good representation of what you would do in real life. In real life you build one model and have one chance to evaluate it. So cross validation as I explained in (1) is akin to simulating 10 different models, while your suggestion here is just as computationally expensive but only gives you 1 simulation of a real-life scenario. If you follow the process I laid out in (1), then generally you average across the 10 confusion matrixes of the test split against your model.

  3. I think we've covered how this isn't best practice, and I'm not sure I entirely understand your suggestion here. Feel free to comment.

$\endgroup$
2
  • $\begingroup$ @TannerPhilips The k different models option with averaging I have seen explained here: link, here (second paragraph): link and here: link. Also in neural networks there could be a validation split that is used for testing upon when fitting each batch of data to get an accuracy and see if it changes so we know when to stop training. (this is different from cross-validation) $\endgroup$ – kosatka Oct 23 '20 at 15:07
  • $\begingroup$ Everything you posted makes sense to me. I'm not familiar with those other methods, but hopefully what I said about cross-validation is helpful. $\endgroup$ – Tanner Phillips Oct 24 '20 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.