1
$\begingroup$

For a dataset that is not too large, I am trying a couple of models for prediction. I get the following train and test MSE for them:

  • Model 1: Train MSE = 100, Test MSE = 104
  • Model 2: Train MSE =   30, Test MSE =   65

Now the second model has obviously a smaller test error. Should I just forget about the overfitting and variability in it and choose model 2? Are there any other considerations that I should take into account?

Another thing that I was wondering: if I use cross validation can I just compare “cross validation test score means” and not care about the train and test score difference?

$\endgroup$
0
$\begingroup$

Now the second model has obviously a smaller test set. Should I just forget about the overfitting and variability in it and choose model 2?

If you tested the two different models on two different test sets, you probably can't conclude anything at all; you would need to test the models on the same test set in order to learn which model makes better predictions on a held out set.

Another thing that I was wondering: if I use cross validation can I just compare "cross validation test score means" and not care about the train and test score difference?

If you're primarily interested in prediction and are optimizing for one thing, I'd rather have a model with a low test error than a model with similar train and test errors. For example, in the case where a model is absolutely useless for prediction, it will have similar train and test errors (both errors will be large because the model is bad), but the model will not be of any practical use at all.

$\endgroup$
1
  • $\begingroup$ Just edited my question. I meant "smaller test error", not "smaller test set". Thanks for your answer. $\endgroup$
    – sam
    Apr 22 at 20:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.