2
$\begingroup$

I know the differences between training, validation and test data. Just to be clear

  • "training data" is used to make multiple models with different hyperparameters
  • "validation data" is used to choose one of these models
  • "test data" is used to ascertain how well the chosen model is really doing

or to quote "Ripley, B.D. (1996) Pattern Recognition and Neural Networks, Cambridge: Cambridge University Press, p. 354":

Training set: A set of examples used for learning, that is to fit the parameters [i.e., weights] of the classifier.

Validation set: A set of examples used to tune the hyperparameters [i.e., architecture, not weights] of a classifier, for example to choose the number of hidden units in a neural network.

Test set: A set of examples used only to assess the performance [generalization] of a fully-specified classifier.

Now let's assume I am already done with the whole modelling process, I got my model and have put it into production (e.g. to classify products on an assembly line in a manufacturer). What do I call the data that is now classified? I am not using it for testing, as I am not getting any model statistics from it - it does not have labels - but I am applying my model to it.

If there is not already a word for it, and idea might be classificand, modelland or something similar, since "-end" or "-and" are suffixes forming nouns denoting patients or recipients of actions, such as addend, subtrahend, and dividend., but I was actually going more for something like "... data".

PS: rand on Why is it called "validation" dataset? I feel that is misleading name. "tuning" dataset would be more appropriate, because that is what it is used for. The "test data" is currently used for actual validation. rand off

Improve this question
$\endgroup$
4
  • $\begingroup$ I think "validation" vs "test" is just one of those cases where near-synonyms in everyday English are used to make a rather important distinction in technical jargon (e.g. "probability" vs "likelihood"). If their meanings were swapped round we wouldn't be any worse off. machinelearningmastery.com/difference-test-validation-datasets gives a pretty good account of their usage. $\endgroup$
    – Scortchi ♦
    Commented Sep 15, 2017 at 11:10
  • $\begingroup$ @Scortchi:Not quite sure why the blog says "Generally, the term “validation set” is used interchangeably ..." when it is clear from the same post, that they are not interchangeable. Besides that quotes from the blog define the sets the same way as I did. I still hate the terms, as I have for years now, except for "training" that is ok ;-). $\endgroup$
    – Make42
    Commented Sep 15, 2017 at 11:34
  • $\begingroup$ "Generally, the term 'validation set' is used interchangeably with the term 'test set' and refers to a sample of the dataset held back from training the model." - that makes sense. It's only with reference to a particular three-stage approach that you need to make the "validation"/"test" distinction. $\endgroup$
    – Scortchi ♦
    Commented Sep 15, 2017 at 13:33
  • $\begingroup$ @Scortchi: If you have a two-step process because of k-fold cross validation, you don't have the validation/test data to be the same, but validation/training data to be the same, don't you? $\endgroup$
    – Make42
    Commented Sep 15, 2017 at 14:26

2 Answers 2

Reset to default
2
$\begingroup$

There doesn't seem to be a special word for it but I would argue that it isn't needed either. After all, the terms train,validation and test data(sets), describe subsets of the data you use during the Train-and-Test-Procedure of modelbuilding. Whatever you feed into your model afterwards, doesn't really need a defined name, because it is outside of this procedure and your initial dataset. I would argue, it doesn't need to be further differentiated and can just be called "data".

But this whole question seems to be mostly opinion based and could be marked as such.

Ps: regarding validation dataset: I think the name comes from the idea that you use this set to chose which algorithm you want to use. So you validate the performance of different candidates. But because it's more commonly used to tune the meta-aparameter of an algorithm, tuning-dataset would be a good name to.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Disagree: It is not opinion based: I asked whether there is such a name. That is a yes/no question :-/, not opinion. $\endgroup$
    – Make42
    Commented Sep 15, 2017 at 9:39
  • 1
    $\begingroup$ Okay, then I think the answer is just " No". The second part of your question did sound to me like you were asking for our opinions on the proposed words $\endgroup$
    – Bobipuegi
    Commented Sep 15, 2017 at 9:43
1
$\begingroup$

You are right that the word 'validation' is used in a lot of ways (https://en.wikipedia.org/wiki/Validation), but the most basic 'thesaurus'-definition for 'to validate' is 'to ascertain the truth or authenticity of something'. So if you're goal is to check whether your model doesn't only perform well in you're own dataset (whether the training or testing part), but on data collected by others, based on slightly different definitions, or any other setting than the development one in general, I feel 'validation' is quite spot-on.

In biomedical prognostic or diagnostic prediction model research these semantics are indeed applied as such: applying a previously developed model to new unseen data and checking the performance is called a 'validation study', performed on respectively a validation dataset. Admittedly, in these studies we often use internal validation to denote techniques as cross-validation or bootstrapping (i.e. setting aside a part of your data for training and another for testing), and external validation to denote testing the performance in a completely new or other sample/setting.

To conclude along these lines, you might want to call your new data an 'external validation dataset'.

(!) Do note, this is all based on my assumption you actually mean to test the performance in new data! If for example you want to compare using your model to make a decision against human insight to make the same decision, this would be an 'impact'study. In such a study, the model's predictions and its external validity are not tested, but rather its impact on another outcome (costs, health improvement, etc.)

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ A validation dataset is not used to "ascertain the..." anything, it is used to choose a model out of many. What you use for a ascertaining how well the model is working, one uses the test dataset. (Please check out my edit to my question.) I feel you switched the meaning of "validation" and "test". "test" might be what you called "external validation". However, in every case: train, validation, test data always have labels in supervised learning. I am searching for a name regarding data that has no labels, which the model is applied with intend of gaining actually new insights. $\endgroup$
    – Make42
    Commented Sep 15, 2017 at 9:50
  • $\begingroup$ That is a terminology not shared in biomedical research, which I referred to. To 'validate' my use of validation see for example the TRIPOD statement (equator-network.org/reporting-guidelines/tripod-statement) which is the leading reporting standard for biomedical studies presenting prediction models (for diagnostic/prognostic purposes). I know there are different schools of using these words (i.e. your use of training/testing/validating is probably more machine learning orientated). But if that is your take then IMHO @bobipuegi 's answers are correct: it's a matter of opinion. $\endgroup$
    – IWS
    Commented Sep 15, 2017 at 11:55
  • $\begingroup$ If anything, using 'external validation dataset' (note the emphasis on external) might give you what you want. $\endgroup$
    – IWS
    Commented Sep 15, 2017 at 12:14
  • $\begingroup$ Just for clarification: Does your "external validation dataset" have labels? $\endgroup$
    – Make42
    Commented Sep 15, 2017 at 12:23
  • $\begingroup$ If with labels you mean the dependent variable your model is developed to predict/classify, then yes. This would be necessary in order to measure the performance of the model within the 'external validation' dataset. $\endgroup$
    – IWS
    Commented Sep 15, 2017 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.