Why is it that my colleagues and I learned opposite definitions for test and validation sets?

Question

In my master's program I learned that when building a ML model you:

1. train the model on the training set
2. compare the performance of this against the validation set
3. tweak the settings and repeat steps 1-2
4. when you are satisfied, compare the final model against the test (hold out) set

When I started working as a DS I raised a question as to the size of the test and validation sets, because it looked as though someone had labeled them wrong. This caused confusion because apparently everyone else used the "test" set in step 2 and held out the "validation" set for step 4.

I assumed I had learned it wrong and no harm was done because I just switched the terms to be consistent. However I was restudying some deep learning books and noticed that according to the creator of Keras, I was right all along! Just before I wrote this question I found this one that suggests the OTHER definition of test/validation sets are correct...

Is this something that is agreed upon? Is there a divide among the classical ml method and deep learning practitioners as to what the correct terms are? As far as I can tell nobody has really discussed how some statisticians/data scientists use completely opposite definitions for the two terms.

Answer 1

For machine learning, I've predominantly seen the usage OP describes, but I've also encountered lots of confusion coming from this usage.

---

Historically, I guess what happened (at least in my field, analytical chemistry) is that as models became more complex, at some point people noticed that independent data is needed for verification and validation purposes (in our terminology, almost all testing that is routinely done with models would be considered part of verification which in turn is part of the much wider task of method validation). Enter the validation set and methods such as cross validation (with its original purpose of estimating generalization error).

Later, people started to use generalization error estimates from what we call internal verification/validation such as cross validation or a random split to refine/optimize their models. Enter hyperparameter tuning.
Again, it was realized that estimating generalization error of the refined model needs independent data. And a new name was needed as well, as the usage of "validation set" for the data used for refining/optimizing had already been established. Enter the test set.

Thus we have the situation where a so-called validation set is used for model development/optimization/refining and is therefore not suitable any more for the purpose of model verification and validation.

---

Someone with e.g. an analytical chemistry (or engineering) background will certainly refer to the data they use/acquire for method validation purposes as their validation data* - and that is correct usage of the terms in these fields.

*(unless they know the different use of terminology in machine learning, in which case they'd usually explain what exactly they are talking about).

---

Personally, in order to avoid the ongoing confusion that comes from this clash of terminology between fields, I've moved to using "optimization data/set" for the data used for hyperparameter tuning (Andrew Ng's development set is fine with me as well) and "verification data/set" for the final independent test data (the testing we typically do is actually verification rather than validation, so that avoids another common mistake: the testing we typically do is not even close to a full method validation in analytical chemistry, and it's good to be aware of that)

Another strategy I find helpful to avoid confusion is moving from splitting into 3 data sets back to splitting into training and verification data, and then describing the hyperparameter tuning as part of the training procedure which happens to include another split into data used to fit the model parameters and data used to optimize the hyperparameters.

Answer 2

Apparently, the terms are used ambiguously, but I always seen them used as that there are three (or more) sets of data: train set used for training the model, validation set for assessing the performance of the model when tuning it, and held-out test set that you use at the very end to assess the performance of the model. These names are used in Google's Machine Learning Crash Course, the Deep Learning with Python book by François Chollet, the Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow book by Aurélien Géron, The Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani, and Jerome Friedman, and a number of other books.

If you find this naming convention confusing, you can, as Andrew Ng, use the train/dev/test naming, where the dev set is used for development.

Answer 3

I was taught that you have a train/test split for tuning then you have a validation set to 'validate' that you haven't overfitted your test split. If you have a small dataset then you just have your train/test split, I would never call it a train/validation split because I think of validation as the final step to 'validate' all of your results, whereas test is to 'test' your model on unseen data. But you could easily flip them and it's all the same!

I have noticed the terms used back and forth but it doesn't really matter what you call it as long as everyone is on the same page.

EDIT after some digging:

Your usage is the correct usage although it is known that the flip side is frequently used (although incorrectly). Wiki even has a section reviewing this discrepancy.

Pure conjecture but I think it most likely stems from this:

Where if you just have a simple split it is train/test and this split used to be a standard way to tune for simple models so the 'test' set was everything.

And to add further to this, it seems if you only do 5 fold cross validation then you are doing 5 train sets and 5 test sets. BUT if you then add a third holdout set then you now have 5 train sets, 5 validation sets, and 1 test set.