machine-learning : Why training set and test set need to be independent and identically distributed?

Question

My machine-learning book that I'm reading only says that they need to be but not why? My intuition says that if they are that leads to a better learning, if they were not it would be like we are training and testing on the same set? Am I right ?

Answer 1

There are many issues related to your question. The first is to ask whether your sample size is large enough for split-sample validation to be reliable. Typically this requires 20,000 or more observations, otherwise the "luck of the split" will be an important issue. In many cases, internal validation is more meaningful and precise (see here for more information). More directly to your question, the nature of any independent test sample that you think is needed will depend on what you are trying to do. If you are measuring overfitting by seeing how much predictions fall apart when applied to independent data from the identical data stream as used when building the model, then you want to test data to come from the same population of subjects. Other researchers will want to show that predictions in one setting transport to another setting, even when the way variable are measured changes. For that situation, the test sample may be different in fundamental ways.