How do you refer to data that's not part of train/test/validation?

Question

The purpose of creating a machine learning model is to deploy in live situation, e.g., classification in the wild.

The model is trained on the training set, tuned on the validation set, and evaluated on the test set.

When it is being used in the wild, how do practitioners typically refer to data that is fed into the model during deployment? Obviously this data is typically more general than the data in train/test/validation sets.

I had always assumed this term is called "out of sample" data...but it seems that there is some conflict between this terminology and test data.

Answer 1

I'm going to assume that you encountered some ambiguity relative to that, and that you want a solution to clarify it in the future.

In the context of prediction, "new observations", "new data", and "unseen data" are common expressions to refer to observations on which the model has not been trained (yet). This is not entirely satisfying relative to your question, but I'm getting there.

If you train a model on all your sets, then these expressions refer to what you described, i.e. data from your population of interest that haven't been collected. However, if you trained a model only on the "train set", you could call observations from the test sets "new observations" too, relative to this training model.

That's why there might be some ambiguity or misunderstanding relative to these "new observations", but only if you don't specify if you're talking about the intermediate "training model" or about the final model (that you should train on all your available data).

So it raises the question: In what kind of situation wouldn't you specify if you're talking about an "intermediate" model or about a final model? If you can specify it, then this is the solution to clarify the ambiguity of the terms "new observations", "new data", etc. For example:

The final model failed to predict new observations well.

corrects the possible ambiguity of

The model failed to predict new observations well.

Yet, you may be in a context where you are not talking about models at all, but about the data collection process. In this case, if you want a specific expression to refer to data that have not been collected, there is some terminology from the survey methodology literature that you could use.

However, the reasons why data have not been collected may vary:

• Either units have not been selected during the sampling process, due to chance (e.g. if you randomly sample 10% of your population, then 90% are not selected). You could call them non-selected units.
• Or they have been selected during the sampling procedure, but you unfortunately failed to gather any data about them: unit non-response or total non-response.

From a methodological point of view, these two situations may be treated differently, relative to re-computing survey weights, trying to collect data again for selected units that did not respond, etc. That's probably why, to the best of my knowledge, there's not an umbrella word to encompass both situations in a survey methodology context. But is not having an umbrella term really a problem anyway?

Answer 2

Obviously this data is typically more general than the data in train/test/validation sets.

This can be true, but often we assume that our samples are relatively indicative of the real world.

If you don't have enough samples you can either generate new samples, through augmentation or something more sophisticated; or, from your real world inferences, you can use active learning to help identify and tamp down outliers. For the former case you usually call it "synthetic data" and in the latter case usually just "new data" until it's incorporated into the model.

If you're just running plain inference and you are happy with the model, I usually hear "real (world) data." But, I personally think it's a really bad name because (hopefully) your training data is also real world data, making the term misleading. At the very least it should be "new real (world) data." To be fair, I do also hear "new [context] data," where [context] can be user or camera or whatever real world measurement context the model is applied to.