Q: Consider two linear models, and a dataset split into a training set, a validation set, and a test set in a 70-15-15% proportion. The two models produce a comparable and low mean squared error (MSE) over the training set and comparable but high MSE over the test set. Over the validation set, the MSE committed by the first model is quite high, unlike the second model. Are the following statements true:

  1. The test set is distributed differently than the rest of the dataset
  2. The first model overfits over the training set
  3. The third model overfits over the training set

I thought that 2. and 3. must be true, regardless of what is happening over the validation set. How would you infer the distribution of the test set from such information? I believe I can't say anything about it.

  • $\begingroup$ Statement 3 is false because there is no third model. $\endgroup$
    – Ceph
    Oct 12 at 11:16
  • $\begingroup$ I assumed a typo. $\endgroup$
    – marco
    Oct 12 at 11:18

Excepting corner cases such as a single outlier causes the very high MSE in test data, (1) is also true. If validation and test set had similar distribution (= the population distribution), MSE would have been the same on them.

  • $\begingroup$ That exception seems pretty important. $\endgroup$
    – Dave
    Nov 9 '20 at 23:04
  • $\begingroup$ NO, (1) is not necessarily true. It could of course be the case, but we cannot conclude this from the observations we have without knowing far more details about the situation. $\endgroup$ Nov 10 '20 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.