Do everything on the training data.

During model development, act like the test data do not exist.

Consider how machine learning is used for products like Siri's speech recognition. The goal is to make a prediction about speech that Siri hasn't heard. In fact, that bit of sound has not even occurred. Engineers couldn't possibly include such data into their model development steps, yet the expectation is that the model will have some level of performance on data Siri has never encountered.

Having a test set is a simulation of this where you hide the data from the model being developed.

Edit: As whuber pointed out, this is for figuring out what kind of model you want to use. Once you decide that you’re using model X for the production version, then your entire data set would be used as you’ve made the decision that it is reliable enough to make decisions where you do not know the correct answer. Those new observations start to function as your out-of-sample data. After all, if you had some stock price predictor do great on cross validation but suddenly start losing tons of money when it got used for real, you’d go back to tweaking your model.

Do everything on the training data.

(See edit.)

During model development, act like the test data do not exist.

Consider how machine learning is used for products like Siri's speech recognition. The goal is to make a prediction about speech that Siri hasn't heard. In fact, that bit of sound has not even occurred. Engineers couldn't possibly include such data into their model development steps, yet the expectation is that the model will have some level of performance on data Siri has never encountered.

Having a test set is a simulation of this where you hide the data from the model being developed.

Edit: As whuber pointed out, this is for figuring out what kind of model you want to use. Once you decide that you’re using model X for the production version, then your entire data set would be used as you’ve made the decision that it is reliable enough to make decisions where you do not know the correct answer. Those new observations start to function as your out-of-sample data. After all, if you had some stock price predictor do great on cross validation but suddenly start losing tons of money when it got used for real, you’d go back to tweaking your model.

Do everything on the training data.

During model development, act like the test data do not exist.

Consider how machine learning is used for products like Siri's speech recognition. The goal is to make a prediction about speech that Siri hasn't heard. In fact, that bit of sound has not even occurred. Engineers couldn't possibly include such data into their model development steps, yet the expectation is that the model will have some level of performance on data Siri has never encountered.

Having a test set is a simulation of this where you hide the data from the model being developed.

Do everything on the training data.

During model development, act like the test data do not exist.

Consider how machine learning is used for products like Siri's speech recognition. The goal is to make a prediction about speech that Siri hasn't heard. In fact, that bit of sound has not even occurred. Engineers couldn't possibly include such data into their model development steps, yet the expectation is that the model will have some level of performance on data Siri has never encountered.

Having a test set is a simulation of this where you hide the data from the model being developed.

Edit: As whuber pointed out, this is for figuring out what kind of model you want to use. Once you decide that you’re using model X for the production version, then your entire data set would be used as you’ve made the decision that it is reliable enough to make decisions where you do not know the correct answer. Those new observations start to function as your out-of-sample data. After all, if you had some stock price predictor do great on cross validation but suddenly start losing tons of money when it got used for real, you’d go back to tweaking your model.