ref: Figure 1: overfitting and underfitting

Shouldn't cross validation error follow training error and remain low ? Is this because the cross validation data set is smaller than training data set? Overfitting by definition means the model is fitting perfectly and produces expected result and thus the error is supposed to be low or none. What am i missing?

  • $\begingroup$ Your image is not working (you probably should not put spaces in filenames). $\endgroup$ – Matthew Drury Aug 7 '18 at 22:56
  • $\begingroup$ No overfitting means that you are using variables that don't contribute but make the fit during training look better because you are fitting to the noise. When you apply cross validation you are coming closer to exhibiting the true error. So the error expressed by MSE is larger. $\endgroup$ – Michael R. Chernick Aug 7 '18 at 23:00
  • 1
    $\begingroup$ I removed the image and attached a link. Thanks Matthew. $\endgroup$ – user2891264 Aug 7 '18 at 23:01
  • $\begingroup$ Thanks @MichaelChernick Since the error is high, shouldn't it be called underfitting? Is MSE an abbreviation for Model Selection Error? $\endgroup$ – user2891264 Aug 7 '18 at 23:14
  • $\begingroup$ MSE is mean squared error. It is called overfitting because the fit based on all the training data includes too many variables and makes the fit look too good. $\endgroup$ – Michael R. Chernick Aug 8 '18 at 0:14

Sorry, my rep is too low to comment so will be posting as an answer.

The benefit of conducting CV is that you can train your model over the entire data that you have and yet still be able to get a good estimate of the true error of your model.

The more variables you include in your model, the lower the training error will get. However, doing so results in overfitting because your model becomes too specialized to its training data that when unseen data comes along it will instead perform worse. As Michael said, this is due to the model, in order to minimize training error, ends up fitting to the noise present in the data. When you then try to predict unseen data which will have a different noise signature using the model, you will end up getting a greater prediction error.

CV simulates this environment by holding out data for test purposes. This plays the role of the unseen data. CV then does this K times and averages the error as the validation error. Hence the validation error increases if the model is overfitted.

  • $\begingroup$ The picture had a circle for overfitting covering both training error curve and the cross validation error curve which was my source of confusion. Fitting to the noise in data, interesting perspective, something to think about. $\endgroup$ – user2891264 Aug 8 '18 at 14:26
  • $\begingroup$ The circles are there to represent the 2 different scenarios. On the left is the typical trend when the model is overfitted. This results in low training error but high validation error (which is the error when you use the model to predict an unseen data set). On the right is the converse case when the model is underfitted. It will have high training error AND validation error because the model is simply insufficient in predicting the response variable. $\endgroup$ – legohorse Aug 9 '18 at 5:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.