I got two generalized linear model M1 and M2. I split my data into train (50%) and test (50%) and I compute the MSE on each set. I got :

On train set, MSE(model M1) > MSE(model M2)

On test set, MSE(model M1) < MSE(model M2)

How can I choose the best model ?

  • 1
    $\begingroup$ Did you tried to shuffle observations in training/test set and measure performance again? $\endgroup$
    – Slam
    Jun 22, 2017 at 13:10
  • $\begingroup$ Yes but I got the same result. Which model should I choose ? $\endgroup$
    – user44677
    Jun 22, 2017 at 13:17
  • 4
    $\begingroup$ Imagine you're teaching a subject and you have two students, Student A and Student B. You have two exams, Exam 1 and Exam 2. Exam 1 is an old exam that has both its questions and answers posted online, while Exam 2 has no such information available. You know that both students A and B consulted that posted exam to study. The next day you make them do Exam 1. The day after, you make them do Exam 2. Student A does better than Student B on Exam 1. Student B does better than Student A on Exam 2. From this information, which of the students do you think better understands the material? $\endgroup$ Jun 22, 2017 at 13:30
  • $\begingroup$ @Bridgeburners Awesome explanation, I understand know $\endgroup$
    – user44677
    Jun 22, 2017 at 15:21

1 Answer 1


You should choose M1 as it is generalizing better than M2 on data that has not been used for training. Probably M2 is overfitting, that's why it gives better results on data that it has already seen, but worse results in data that it has never been exposed to.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.