I found somewhere that test set must not be used as a validation set. Why?

Validation set is acted upon when the model parameters are fixed, and learning happens only through backprop on the training batch.

So, why can't I use test data as validation data?

  • 8
    $\begingroup$ My client just mailed out ads in four different colors of envelopes, 1000 of each color. The returns on the orange envelopes were 12 out of 1000, whereas the returns on the other three colors were between 8 and 11 out of 1000. My model therefore predicts that orange envelopes get the best returns. I will prove this is a good model by applying it to my data: yes indeed, it's 100% correct; the returns were best on the orange envelopes. Now I can publish my marketing book knowing its advice has been statistically proven. (This is a real example from a real book.) $\endgroup$ – whuber Sep 30 '16 at 17:00
  • 4
    $\begingroup$ Can you define how you are using the terms "test set" & "validation set"? (Different people mean different things by these.) Do you also have a 'training set'? How do data end up in these different sets? Etc. $\endgroup$ – gung Sep 30 '16 at 17:06
  • $\begingroup$ @gung 1000 training data, 1000 test data. I train the model, after each epoch I try the model on validation data = test data. An alternative is 950 training data, 50 validation data, after each epoch I try the model on validation data. What is the difference here? $\endgroup$ – John77 Sep 30 '16 at 17:12
  • $\begingroup$ @john77 I think my answer explains what the difference is - to assess how the model will perform on new data you need to use data that hasn't been used to develop the model (eg in deciding when to stop training) $\endgroup$ – seanv507 Sep 30 '16 at 18:15

I presume you're already understand why performance on the training set isn't representative of the actual performance of the trained model: overfitting. The parameters you learn during training are optimized to the training set. If you're not careful, you can over-optimize the parameters, leading to a model that's really, really good on the training set, but doesn't generalize to the.

The thing is, in practice the "parameters" of the training method aren't the only thing you need to specify for a learning example. You also have hyperparameters. Now, those hyperparameters might be an explicit part of the model fitting (like learning rate), but you can also view other choices as "hyperparameters": do you choose an SVM or a neural network? If you implement early stopping, at what point do you stop?

Just like overfitting of the parameters on the training set, you can overfit the hyperparameters to the validation set. As soon as you use the results of the method on the validation set to inform how you do modeling, you now have the chance of overfitting to the training+validation set combo. Perhaps this particular validation set does better with an SVM than the general case.

That's the main reason people separate out the validation and test sets. If you use a set during your model fitting - even at the "hmm, that method doesn't do so well, maybe I should try ..." level - the results you get on that set will not be fully indicative of the general results you'll obtain on completely new data. That's why you hold out a fraction of the data till the very end, past the point where you're making any decisions on what to do.


I will stick to neural networks. Note that some people swap validation and test set around.

In neural nets you train until your performance on the validation set starts getting worse ('stopped training'). Therefore the weights are being influenced by the validation set (just not through backprop).

Now you test your network with brand new data (the test set) which has not been used in the model building process at all. This same approach applies also when selecting hyperparameters eg dropout level/L2 regularisation parameter etc.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.