Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a question regarding the Cross-validation process. I am in the middle of a course of the Machine Learning on the Cursera. One of the topic is about the Cross-validation. I found it slightly difficult to follow. I do know why we need CV because we want our models to work well on future (unknown) data and CV prevents from overfitting. However, the process itself is confusing.

What I have understood is that I split data into 3 subsets: training, validation, and test. Train and Validation is to find optimum complexity of a model. What I don't understand is the third subset. I understand I take a number of features for the model, train it and validate it on Validation subset and look for the minimum Cost Function when I change the structure. When I found it, I do test the model on Test subset. If I have already found minimum Cost Function on Validation subset, why would I need to test it again on Test subset ???

Could someone please clarify this for me?

Thank you

share|improve this question
    
You will find the paper and discussion in [1] not irrelevant to your question. From memory, it mentions three subsets: the first to formulate a model, the second to estimate its parameters and the third to validate it by prediction. [1] Chatfield, C. Model Uncertainty, Data Mining and Statistical Inference (With discussion) Journal of the Royal Statistical Society. Series A (Statistics in Society), 1995, 158, 419-466 –  Hibernating Dec 17 '13 at 15:55
    
    
Cheers! The thread behind this link is also a good source for understanding this :) –  ZikO Dec 19 '13 at 14:27

3 Answers 3

up vote 17 down vote accepted
  • The training set is used to choose the optimum parameters for a given model. Note that evaluating some given set of parameters using the training set should give you an unbiased estimate of your cost function - it is the act of choosing the parameters which optimise the estimate of your cost function based on the training set that biases the estimate they provide. The parameters were chosen which perform best on the training set; hence, the apparent performance of those parameters, as evaluated on the training set, will be overly optimistic.
  • Having trained using the training set, the validation set is used to choose the best model. Again, note that evaluating any given model using the validation set should give you a representative estimate of the cost function - it is the act of choosing the model which performs best on the validation set that biases the estimate they provide. The model was chosen which performs best on the validation set; hence, the apparent performance of that model, as evaluated on the validation set, will be overly optimistic.
  • Having trained each model using the training set, and chosen the best model using the validation set, the test set tells you how good your final choice of model is. It gives you an unbiased estimate of the actual performance you will get at runtime, which is important to know for a lot of reasons. You can't use the training set for this, because the parameters are biased towards it. And you can't use the validation set for this, because the model itself is biased towards those. Hence, the need for a third set.
share|improve this answer
    
I can see this now. Thank you. I did not think about the fact I'd need one more subset to find unbiased performance in terms of the model structure. Cheers :) It's very good explanation. –  ZikO Dec 17 '13 at 14:32

If I have already found minimum Cost Function on Validation subset, why would I need to test it again on Test subset

Because of random error: Usually you only have a finite number of cases.

Optimization of the validation (inner test) performance means that you may be overfitting to that inner test set. The inner test set contributes to the estimation of the final model and is thus not independent of the model.

This means that you need to have another (outer) test set that is independent of the whole modeling procedure (including all optimization and data-driven pre-processing or model selection processes) if you want to estimate the generalization properties.

I recommend that you make a simulation and compare the three different error estimates you can have

  • resubstitution: prediction of the train set
    measures goodness-of-fit
  • inner test (in your nomenclature: validation) set: the quality the optimizer thinks is achieved
  • outer test set: generalization error, independent of model training.

In a simulation you can easily compare them also to a proper, large, independently generated test set. If the set-up is correct, the outer test should be unbiased (w.r.t. the surrogate model it evaluates, not w.r.t. a "final" model built on the whole data set). The inner test is usually optimistically biased, and resubstitution even more optimistically biased.

In my field, the inner test would easily underestimate the generalization error by a factor of 2 - 5 (much more for aggressive optimization schemes).


Note: the nomenclature of the sets is not universal. In my field (analytical chemistry), validation would usually mean the proof of the performance of the final procedure - thus more what your "test" set does than what your "validation" set does.

I therefore prefer to speak of the inner and outer test sets, or of the optimization test set (= inner test set) and then validation set would mean the outer test set.

share|improve this answer
    
I've got the point as well. Thanks for this :) –  ZikO Dec 17 '13 at 14:52

While training the model one must select meta parameters for the model (for example, regularization parameter) or even choose from several models. In this case validation subset is used for parameter choosing, but test subset for final prediction estimation.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.