Q: Consider two linear models, and a dataset split into a training set, a validation set, and a test set in a 70-15-15% proportion. The two models produce a comparable and low mean squared error (MSE) over the training set and comparable but high MSE over the test set. Over the validation set, the MSE committed by the first model is quite high, unlike the second model. Are the following statements true:

  1. The test set is distributed differently than the rest of the dataset
  2. The first model overfits over the training set
  3. The second model overfits over the training set

I thought that 2. and 3. must be true, regardless of what is happening over the validation set. How would you infer the distribution of the test set from such information? I believe I can't say anything about it.

EDIT: corrected an assumed typo

  • $\begingroup$ Statement 3 is false because there is no third model. $\endgroup$
    – Ceph
    Oct 12, 2021 at 11:16
  • $\begingroup$ I assumed a typo. $\endgroup$
    – marco
    Oct 12, 2021 at 11:18
  • $\begingroup$ Unless you are able to offer a correction for the typo (please just edit this into the question), I think we will need to close this question as ambiguous. $\endgroup$
    – Ben
    Oct 12, 2022 at 6:32
  • $\begingroup$ Just edited the question, thank you. $\endgroup$
    – marco
    Oct 13, 2022 at 9:43

2 Answers 2


Excepting corner cases such as a single outlier causes the very high MSE in test data, (1) is also true. If validation and test set had similar distribution (= the population distribution), MSE would have been the same on them.

  • $\begingroup$ That exception seems pretty important. $\endgroup$
    – Dave
    Nov 9, 2020 at 23:04
  • $\begingroup$ NO, (1) is not necessarily true. It could of course be the case, but we cannot conclude this from the observations we have without knowing far more details about the situation. $\endgroup$ Nov 10, 2020 at 12:18
  • $\begingroup$ Hi Paman, you are right, all 3 are true, but may I suggest that you write your answer a bit more clearly (e.g. starting with all 3 are true ...)? $\endgroup$ Nov 15, 2022 at 13:06

I don't think we can say for certain without digging some more, but I suspect the interviewer was looking for this line of thought:

1 and 2 are true, 3 is false.
The validation scores being different is what tips it. The second model still performs well, so we expect the validation set is distributed similarly to the training set, and so the first model is overfit but the second is not. Then since the second model also performs poorly on the test set, we suspect the test set is differently distributed. (Your line of reasoning may be refuted by saying that 1 is true.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.