42
$\begingroup$

In my master's program I learned that when building a ML model you:

  1. train the model on the training set
  2. compare the performance of this against the validation set
  3. tweak the settings and repeat steps 1-2
  4. when you are satisfied, compare the final model against the test (hold out) set

When I started working as a DS I raised a question as to the size of the test and validation sets, because it looked as though someone had labeled them wrong. This caused confusion because apparently everyone else used the "test" set in step 2 and held out the "validation" set for step 4.

I assumed I had learned it wrong and no harm was done because I just switched the terms to be consistent. However I was restudying some deep learning books and noticed that according to the creator of Keras, I was right all along! Just before I wrote this question I found this one that suggests the OTHER definition of test/validation sets are correct...

Is this something that is agreed upon? Is there a divide among the classical ml method and deep learning practitioners as to what the correct terms are? As far as I can tell nobody has really discussed how some statisticians/data scientists use completely opposite definitions for the two terms.

$\endgroup$
7
  • 7
    $\begingroup$ I don’t read the linked question as supporting the other definition, and I’ve never seen the usage you’ve said your colleagues use. $\endgroup$ May 24, 2021 at 15:06
  • 3
    $\begingroup$ I've heard them used in both ways too, and it mostly doesn't matter, but +1 in case any answerer can provide some historical context. $\endgroup$ May 24, 2021 at 15:15
  • 3
    $\begingroup$ I use them as you do. It is consistent with using cross-validation for model tuning. $\endgroup$
    – Michael M
    May 24, 2021 at 16:51
  • 2
    $\begingroup$ I'm from the "data is rare and expensive" era, so what I heard was that you use "train/valid" to determine parameter settings, "test" to estimate real-world performance for those settings, then you put all the data in a pile and use them to train the actual production learner, using proficiency as a practitioner to make sure the combined set learner is has less or equal loss compared to the train/valid learner. $\endgroup$ May 24, 2021 at 18:19
  • 3
    $\begingroup$ @JasonGoemaat What you've written is certainly the correct English language, as you correctly explain. It is rather odd, I feel, that the norm seems to be to the 'wrong' way around. Naturally, if everyone's on the same page, it doesn't really matter. But generally I think it's good to use names which reflect the "real world"/"standard English" usage. This makes it much easier for beginners. I remember being confused by this when learning: "I'm not validating anything in the second step but I am in the final step!" This was a genuine source of confusion for me when I started learning $\endgroup$
    – Sam OT
    May 27, 2021 at 9:30

3 Answers 3

24
$\begingroup$

For machine learning, I've predominantly seen the usage OP describes, but I've also encountered lots of confusion coming from this usage.


Historically, I guess what happened (at least in my field, analytical chemistry) is that as models became more complex, at some point people noticed that independent data is needed for verification and validation purposes (in our terminology, almost all testing that is routinely done with models would be considered part of verification which in turn is part of the much wider task of method validation). Enter the validation set and methods such as cross validation (with its original purpose of estimating generalization error).

Later, people started to use generalization error estimates from what we call internal verification/validation such as cross validation or a random split to refine/optimize their models. Enter hyperparameter tuning.
Again, it was realized that estimating generalization error of the refined model needs independent data. And a new name was needed as well, as the usage of "validation set" for the data used for refining/optimizing had already been established. Enter the test set.

Thus we have the situation where a so-called validation set is used for model development/optimization/refining and is therefore not suitable any more for the purpose of model verification and validation.


Someone with e.g. an analytical chemistry (or engineering) background will certainly refer to the data they use/acquire for method validation purposes as their validation data* - and that is correct usage of the terms in these fields.

*(unless they know the different use of terminology in machine learning, in which case they'd usually explain what exactly they are talking about).


Personally, in order to avoid the ongoing confusion that comes from this clash of terminology between fields, I've moved to using "optimization data/set" for the data used for hyperparameter tuning (Andrew Ng's development set is fine with me as well) and "verification data/set" for the final independent test data (the testing we typically do is actually verification rather than validation, so that avoids another common mistake: the testing we typically do is not even close to a full method validation in analytical chemistry, and it's good to be aware of that)

Another strategy I find helpful to avoid confusion is moving from splitting into 3 data sets back to splitting into training and verification data, and then describing the hyperparameter tuning as part of the training procedure which happens to include another split into data used to fit the model parameters and data used to optimize the hyperparameters.

$\endgroup$
5
  • $\begingroup$ This may be the most helpful answer in that it provides an explanation of why the names don't seem to match what is done with the splits. $\endgroup$ May 25, 2021 at 15:13
  • $\begingroup$ Seeing as this discussion centers on terminology, I think it’s worth noting that ‘verification’ has a precise definition in CS related to the soundness and completeness of a logical system (e.g. in formal verification). The usage here is quite different and relates to the process of confirming that the final model is practically fit for the task for which it was developed. $\endgroup$
    – Greenstick
    May 27, 2021 at 18:08
  • $\begingroup$ I'm starting to like the idea of just calling it the hold out set. $\endgroup$ May 27, 2021 at 20:38
  • 1
    $\begingroup$ @Greenstick: in analytical chemistry, verification would show that the system (model, "method") meets the pre-specified figures of merit, e.g. RMSE, sensitivity, specificity are below/above a certain value). Validation shows fitness for purpose in a wider sense, including e.g. whether the specifications which often use surrogate markers are adequate for the task. $\endgroup$ May 28, 2021 at 10:10
  • $\begingroup$ @cbeleitesunhappywithSX Sure makes sense. Was just pointing out the considerable difference in definition, especially because the OP doesn’t mention analytical chemistry (recognize that was in your answer), but does mention statistics, data science, and machine learning — all of which are quite heavily influenced by computer science and the latter of which has a research subdomain that combines ML with formal verification methods. Just trying to add clarity : ) $\endgroup$
    – Greenstick
    May 29, 2021 at 21:41
35
$\begingroup$

Apparently, the terms are used ambiguously, but I always seen them used as that there are three (or more) sets of data: train set used for training the model, validation set for assessing the performance of the model when tuning it, and held-out test set that you use at the very end to assess the performance of the model. These names are used in Google's Machine Learning Crash Course, the Deep Learning with Python book by François Chollet, the Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow book by Aurélien Géron, The Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani, and Jerome Friedman, and a number of other books.

If you find this naming convention confusing, you can, as Andrew Ng, use the train/dev/test naming, where the dev set is used for development.

$\endgroup$
13
$\begingroup$

I was taught that you have a train/test split for tuning then you have a validation set to 'validate' that you haven't overfitted your test split. If you have a small dataset then you just have your train/test split, I would never call it a train/validation split because I think of validation as the final step to 'validate' all of your results, whereas test is to 'test' your model on unseen data. But you could easily flip them and it's all the same!

I have noticed the terms used back and forth but it doesn't really matter what you call it as long as everyone is on the same page.

EDIT after some digging:

Your usage is the correct usage although it is known that the flip side is frequently used (although incorrectly). Wiki even has a section reviewing this discrepancy.

Pure conjecture but I think it most likely stems from this: enter image description here

Where if you just have a simple split it is train/test and this split used to be a standard way to tune for simple models so the 'test' set was everything.

And to add further to this, it seems if you only do 5 fold cross validation then you are doing 5 train sets and 5 test sets. BUT if you then add a third holdout set then you now have 5 train sets, 5 validation sets, and 1 test set.

$\endgroup$
5
  • 8
    $\begingroup$ This is always the way I heard it, too. I think the main point is the names don't matter as long as everyone is on the same page. $\endgroup$ May 24, 2021 at 16:08
  • 4
    $\begingroup$ What you call them matters in as far as you want to communicate or work with others. If they're used to different names, that makes communicating slightly more laborious and potentially confusing. $\endgroup$ May 24, 2021 at 23:36
  • $\begingroup$ @BernhardBarker I would say that as standard practice we should be asking/receiving BOTH 'validation' and 'test' sets. It then is only ever an issue if they are significantly different in accuracy from each other and we can ask for more clarification if needed. And usually we report like the kfold as the CV accuracy and the validation set so everyone is on the same page there. There are many terms in this space where we should be sticklers and I personally don't think this is one. Report both and say what they are and everything will be fine. $\endgroup$
    – Tylerr
    May 24, 2021 at 23:46
  • 2
    $\begingroup$ I think "testing" now comes after "validation". Certainly if you use "cross-validation" it must before the final step as it has leakage into the training and tuning of the model $\endgroup$
    – Henry
    May 25, 2021 at 1:12
  • 1
    $\begingroup$ @Henry: the leakage does not come from using cross validation, it comes from using its result to optimize the model. This is independent of the particular scheme you use for splitting (single split / cross validation / ...). You could use a single train/dev split with an outer cross validation for estimating the final model's generalization error. $\endgroup$ May 25, 2021 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.