19
$\begingroup$

I know that in order to access the performance of the classifier I have to split the data into training/test set. But reading this:

When evaluating different settings (“hyperparameters”) for estimators, such as the C setting that must be manually set for an SVM, there is still a risk of overfitting on the test set because the parameters can be tweaked until the estimator performs optimally. This way, knowledge about the test set can “leak” into the model and evaluation metrics no longer report on generalization performance. To solve this problem, yet another part of the dataset can be held out as a so-called “validation set”: training proceeds on the training set, after which evaluation is done on the validation set, and when the experiment seems to be successful, final evaluation can be done on the test set.

I see that another (third) validation set is introduced which is justified by overfitting of the test set during the hyperparameters tuning.

The problem is that I can not understand how this overfitting can appear and therefore can not understand the justification of the third set.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ I think lots of the confusion here comes from 2 or 3 rather different meanings of the term "validation". Validation in cross-validation is mostly just a name for a splitting-without-replacement procedure. The "validation" set in the cited text would IMHO be better understood as "tuning" or "optimization set". Whereas "validation" e.g. in engineering or also analytical chemistry means to prove that a given final method is fit for its purpose - which is done (simulated, approximated) by the "test" set. (To me, testing sounds much less final than validation...) $\endgroup$
    – cbeleites unhappy with SX
    Aug 27, 2015 at 20:40

4 Answers 4

Reset to default
18
$\begingroup$

Even though you are training models exclusively on the training data, you are optimizing hyperparameters (e.g. $C$ for an SVM) based on the test set. As such, your estimate of performance can be optimistic, because you are essentially reporting best-case results. As some on this site have already mentioned, optimization is the root of all evil in statistics.

Performance estimates should always be done on completely independent data. If you are optimizing some aspect based on test data, then your test data is no longer independent and you would need a validation set.

Another way to deal with this is via nested cross-validation, which consists of two cross-validation procedures wrapped around eachother. The inner cross-validation is used in tuning (to estimate the performance of a given set of hyperparameters, which is optimized) and the outer cross-validation estimates generalization performance of the entire machine learning pipeline (i.e., optimizing hyperparameters + training the final model).

Improve this answer
$\endgroup$
5
  • $\begingroup$ I could be wrong but the test set is not only for tuning hyperparameters but also for comparing different types of techniques like e.g. the performance of SVM compared to LDA or random forest as I indicated in my answer. $\endgroup$
    – user83346
    Aug 26, 2015 at 13:43
  • $\begingroup$ @fcoppens Yes, definitely. I didn't explicitly mention this, but that certainly also fits into that step. $\endgroup$
    – Marc Claesen
    Aug 26, 2015 at 14:56
  • 2
    $\begingroup$ more generally, tuning or optimization here is any kind of data-driven (by the inner cross-validation) process. $\endgroup$
    – cbeleites unhappy with SX
    Aug 27, 2015 at 20:36
  • $\begingroup$ @MarcClaesen I have a question. If I have 2 different values of the hyperparameter, and I fit these 2 models on the training set, then apply them on the test set, then should the error rate I obtained on the test set are unbiased estimator of the true prediction error rate of these 2 models? Because if you now give me another independent test data set, then this new test data set is in essence the same as the previous test set. So if a model has better test score, it should also outperform the other model on a new test data set. Why we need validation set? $\endgroup$
    – KevinKim
    Mar 27, 2017 at 0:51
  • $\begingroup$ "If you are optimizing some aspect based on test data, then your test data is no longer independent and you would need a validation set." that's insightful $\endgroup$
    – Sudip Bhandari
    Jul 5, 2017 at 12:10
11
$\begingroup$

I think it's easiest to think of things this way. There are two things that cross validation is used for, tuning the hyper parameters of a model/algorithm, and evaluating the performance of a model/algorithm.

Consider the first use as part of the actual training of the algorithm. For instance cross validating to determine regularization strength for a GLM is part of establishing the final result of the GLM. This use is typically called internal cross validation. Because (hyper)parameters are still being set, the tuning set loss is not a great measure of the actual algorithms performance.

The second use of cross validation is using data that was held out of the entire process which produced the model, to test its predictive power. This process is called external cross validation.

Note that internal validation may have been part of the process which produced the model so in many cases both internal and external cross validation are necessary.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Personally, I would not use the phrase "external cross validation", as I would see cross validation as the repeated splitting off of different validation sets from the training set for model selection and tuning purposes. You cannot meaningfully do this repeatedly with the test set, as that is as a one-off proxy for future as-yet-unknown data used to judge the performance of the final model. $\endgroup$
    – Henry
    Aug 26, 2015 at 10:11
  • 3
    $\begingroup$ Henry, I don't think you are understanding external cross validation. You can "do this repeatedly with the test set," repeatedly holding out some portion of your full data for test purposes while executing your full training procedure on the rest (which may include internal cross validation). External cross validation is still typically done in folds, and allows for all of the original data to at some point be in the test set. $\endgroup$
    – jlimahaverford
    Aug 26, 2015 at 11:33
  • $\begingroup$ @jlimahaverford why the internal cv does not produce good measure of the actual algorithm performance? I think mathematically it is. Because when you fit your model, say LASSO with specific hyperparameter $\alpha$, then you apply it on another piece of the data (that you didn't use to fit your model) in the internal cv process, then this error rate should be an unbiased estimator of the true prediction error rate for your specific $\alpha$, isn't it? $\endgroup$
    – KevinKim
    Mar 27, 2017 at 0:55
3
$\begingroup$

During model building you train your models on a training sample. Note that that you can train different models (i.e. different techniques like SVM, LDA, Random Forest, ... or the same technique with different values of the tuning parameters, or a mixture).

Among all different models that you trained, you have to choose one and therefore you use the validation sample to find the one with the smallest error on the test sample.

For this 'final' model we still have to estimate the error and therefore we use the test sample.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I would reverse the wording in your second and third paragraphs: I would use the validation set to find the best model and tune its hyperparameters (doing this with several validation sets which partition the training set makes this cross validation) and, once the model has been finalised, then apply it to the test set to see an example of the model's out-of-sample performance. $\endgroup$
    – Henry
    Aug 26, 2015 at 10:16
  • $\begingroup$ @Henry: OK Henry, I think the labels of the phases are less important than thier sequence/contents, but your remark makes sense to me, I will edit the text, thanks (+1), $\endgroup$
    – user83346
    Aug 26, 2015 at 10:42
1
$\begingroup$

Cross-validation does not completely overcome the over-fitting problem in model selection, it just reduces it. The cross validation error depends on the data set you use. The smaller the data set, the higher would be the cross validation error.

Additionally, if you have high degrees of freedom in model selection, then there is a danger of the model performing poorly, as the cross validation criterion gets overfitted.

So, when the data is divided into 2 sets, a.k.a the training and testing sets, the splitting is done statically. So, there is a chance of overfitting the training set. However, the cross validation sets are created through different methods, like the k-fold cross validation, Leave-out-one-cross-validation(LOOCV), etc which helps ensure that the exact fit reward of the 2-set split is eliminated and thus the chance of over fit is reduced.

These are some resources which would help you understand better.

So, cross validation would help you when you have a bigger data set, rather than a smaller one.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.