Q: Consider two linear models, and a dataset split into a training set, a validation set, and a test set in a 70-15-15% proportion. The two models produce a comparable and low mean squared error (MSE) over the training set and comparable but high MSE over the test set. Over the validation set, the MSE committed by the first model is quite high, unlike the second model. Are the following statements true:
- The test set is distributed differently than the rest of the dataset
- The first model overfits over the training set
- The third model overfits over the training set
I thought that 2. and 3. must be true, regardless of what is happening over the validation set. How would you infer the distribution of the test set from such information? I believe I can't say anything about it.