6
$\begingroup$

I am working on a machine learning experiment comparing the use of multiple different neural network classifiers by applying them on a large number of datasets, using stratified 10-fold cross-validation. I measure the performance as the average of the errors on the validation set (sometimes referred to as test set) of the 10-fold cross-validation procedure.

My question is, would it be ok to use this same validation set to do an early stopping of the training procedure? This early stopping would be performed by applying the trained model after each epoch to the validation set and measuring the performance, and if it declines for a number of successive learning epochs, the learning would be halted and we would take the epoch that produced the last good performance. This would be applied to all the different techniques, and across all the different datasets.

Is this ok? Or is it statistically inaccurate?

$\endgroup$
  • $\begingroup$ I think this question is hard to understand, particularly the exact setup OP is using. How many data splits do you have (train/dev/test or only two, or...)? 10xCV on what exactly which of those sets? Which models (or a mixture model?) are you using how for your evaluation (and why does it matter)? $\endgroup$ – fnl Sep 29 '17 at 12:44
  • $\begingroup$ Also, if your question/problem is similar to a very recent question, however, then the answer to your first question is yes. As to the second question, that fully depends on the actual data splits you are working with/referring to (which is also explained [yeah, shameless self-plug... ;-)] in that recent question). $\endgroup$ – fnl Sep 29 '17 at 12:45
6
$\begingroup$

I am not completely clear of what the question is asking, but I think the answer is no. The thing you need to think hard about with cross-validation is that no part of your algorithm can have any access to the test set. If it does, then your cross-validation results will be tainted and not be an accurate measure of the 'true' error.

From your question, I assume you are using some kind of iterative learning algorithm such as GBM and you are using the validation set as a way of determining when your GBM has enough models in its ensemble and has started to overfit. If this is true, then what you are doing is not optimal.

The way to think of this is that the stopping criteria is part of your learning algorithm. If it is part of the algorithm, then it can't use the test set in any way.

You may need to do nested cross-validation. In your outer loop, you divide into test and training sets, then in your inner loop you further divide the training set into sub test and training sets and proceed as you have. The inner loop cross-validation can be used to learn from that training set when to stop the learning, but to get an accurate generalization error you then need to apply that to the test set from the outer loop that hasn't yet been touched by the inner loop whose aim was to find, from the training data, when the best time to stop is. To be clear, say the inner loop cross-validation found that the best number of iterations was 10. In your outer loop you learn a model using the full outer loop training set, iterating 10 times, then see how that performs on the test set.

Does this make sense?

Note that depending on the models in use and the dataset, this may or may not be a big issue. The downside is that nested cross-validation can be very computationally expensive. Doing things the way you have been may well be an appropriate trade-off between accuracy and computational time in your circumstance. The most rigid answer to your question is no, it is not completely valid cross-validation. Whether it is passable for your circumstances is a different question.

$\endgroup$
  • $\begingroup$ Thank you very much for your answer. I am using Neural Networks, but I was trying to keep the question as general as possible, although maybe in doing so I made it ambiguous. I do early stopping to prevent the network from over-fitting the training data. Considering that the purpose of the experiment is to compare different techniques (network configurations in my case), not reach a conclusion about the performance of a single technique on a specific dataset, would it be a sufficient compromise to just do the early stopping using the test set, instead of doing costly nested cross-validation? $\endgroup$ – Islam El-Nabarawy Apr 18 '13 at 9:26
  • $\begingroup$ You still have the fundamental issue that the stopping criteria is part of the algorithm, so shouldn't have access to the test set. When comparing different techniques, some techniques are more prone to over-fitting than others. If you follow your procedure to compare models, you will artificially help these over-fitting prone models in a way that can't be replicated in a real application. That may give you a misleading comparison. As I say, how significant the difference would be depends entirely on your dataset and models. $\endgroup$ – Bogdanovist Apr 18 '13 at 23:45
1
$\begingroup$

The answer is yes as long as you reserve a test set for scoring in your cross validation. This is typically done by a three way partitioning versus a two way: one for train, one for validation (stop), and one for score (test). The validation set is being used to estimate the error on the test set. Thus the idea is that this stratagem gives the minimum error on the test set. Note that there is no way to inhibit over fitting unless you use the validation set to make your stopping decision, where you have achieved your parameter values by using the training set.

$\endgroup$
  • $\begingroup$ Thank you for your response. This is what we ended up doing, combined with a "nested" procedure as suggested by the accepted answer. $\endgroup$ – Islam El-Nabarawy Dec 4 '14 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.