Overfitting is when we have a model which has memorized the training data and does not perform well in real-world cases.

Okay, say that I had some training points which look like this:

enter image description here

What if the red curve was the actual 'real-world' relationship. And I found this exact model through a learning algorithm based on those training observations.

Has my model overfit by definition despite it being the 'real-world' relationship? I am assuming yes, but I just want to make sure.



I think it may be useful to rephrase the definition of overfitting to something like:

A model that does not generalize well to real-world cases although it fits the training data well.

As for your example:

  • If the real world looks like the red line there is by definition no overfitting.
  • But at the same time, if the black dots are all real-world test data you have, you probably still cannot prove this: in real-world situations, 10 cases are just not enough to prove that a function of the shown complexity was successfully fit.

  • To give you an idea about one real-world field: in analytical chemistry, a series of 10 concentration steps covering your desired range of analyte concentrations is usually required to show that your method yields linear response.

| cite | improve this answer | |

You should look at your error on the training and on the validation data. If the validation error increases while the training error decreases then a situation of overfitting may have occurred. The best predictive and fitted model would be where the validation error has its global minimum.

| cite | improve this answer | |
  • 1
    $\begingroup$ Agreed. The critical point is that overfitting occurs when performance on your training set gets better at a cost of the model generalizing less well. While in this case it certainly looks like what we see when a model overfits, if the real world curve really does match the one your training produced, then your model generalizes as well as can be. $\endgroup$ – Pat Jun 25 '14 at 16:26
  • $\begingroup$ What if training error is 0 and validation error is 0, have I still overfit by definition? $\endgroup$ – user46925 Jun 25 '14 at 16:28
  • 2
    $\begingroup$ I don't think so, no. Your validation error has not got worse. Take a look at the initial definition you give: "Overfitting is when we have a model which has memorized the training data AND does not perform well in real-world cases". The 'and' there is crucial. Yes, your model has memorized the training set - but it performs excellently on real world cases. So it has not overfit. $\endgroup$ – Pat Jun 25 '14 at 16:52
  • $\begingroup$ So I have a model that has a lot of variance but does not overfit. Am I correct? Usually, variance and overfitting go together (as they say). $\endgroup$ – user46925 Jun 28 '14 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy