Cross-validation including training, validation, and testing. Why do we need three subsets?

Question

I have a question regarding the Cross-validation process. I am in the middle of a course of the Machine Learning on the Cursera. One of the topic is about the Cross-validation. I found it slightly difficult to follow. I do know why we need CV because we want our models to work well on future (unknown) data and CV prevents from overfitting. However, the process itself is confusing.

What I have understood is that I split data into 3 subsets: training, validation, and test. Train and Validation is to find optimum complexity of a model. What I don't understand is the third subset. I understand I take a number of features for the model, train it and validate it on Validation subset and look for the minimum Cost Function when I change the structure. When I found it, I do test the model on Test subset. If I have already found minimum Cost Function on Validation subset, why would I need to test it again on Test subset ???

Could someone please clarify this for me?

Thank you

Answer 1

• The training set is used to choose the optimum parameters for a given model. Note that evaluating some given set of parameters using the training set should give you an unbiased estimate of your cost function - it is the act of choosing the parameters which optimise the estimate of your cost function based on the training set that biases the estimate they provide. The parameters were chosen which perform best on the training set; hence, the apparent performance of those parameters, as evaluated on the training set, will be overly optimistic.
• Having trained using the training set, the validation set is used to choose the best model. Again, note that evaluating any given model using the validation set should give you a representative estimate of the cost function - it is the act of choosing the model which performs best on the validation set that biases the estimate they provide. The model was chosen which performs best on the validation set; hence, the apparent performance of that model, as evaluated on the validation set, will be overly optimistic.
• Having trained each model using the training set, and chosen the best model using the validation set, the test set tells you how good your final choice of model is. It gives you an unbiased estimate of the actual performance you will get at runtime, which is important to know for a lot of reasons. You can't use the training set for this, because the parameters are biased towards it. And you can't use the validation set for this, because the model itself is biased towards those. Hence, the need for a third set.

Answer 2

If I have already found minimum Cost Function on Validation subset, why would I need to test it again on Test subset

Because of random error: Usually you only have a finite number of cases.

Optimization of the validation (inner test) performance means that you may be overfitting to that inner test set. The inner test set contributes to the estimation of the final model and is thus not independent of the model.

This means that you need to have another (outer) test set that is independent of the whole modeling procedure (including all optimization and data-driven pre-processing or model selection processes) if you want to estimate the generalization properties.

I recommend that you make a simulation and compare the three different error estimates you can have

• resubstitution: prediction of the train set
  measures goodness-of-fit
• inner test (in your nomenclature: validation) set: the quality the optimizer thinks is achieved
• outer test set: generalization error, independent of model training.

In a simulation you can easily compare them also to a proper, large, independently generated test set. If the set-up is correct, the outer test should be unbiased (w.r.t. the surrogate model it evaluates, not w.r.t. a "final" model built on the whole data set). The inner test is usually optimistically biased, and resubstitution even more optimistically biased.

In my field, the inner test would easily underestimate the generalization error by a factor of 2 - 5 (much more for aggressive optimization schemes).

---

Note: the nomenclature of the sets is not universal. In my field (analytical chemistry), validation would usually mean the proof of the performance of the final procedure - thus more what your "test" set does than what your "validation" set does.

I therefore prefer to speak of the inner and outer test sets, or of the optimization test set (= inner test set) and then validation set would mean the outer test set.

Answer 3

While training the model one must select meta parameters for the model (for example, regularization parameter) or even choose from several models. In this case validation subset is used for parameter choosing, but test subset for final prediction estimation.