Overfitting is when we have a model which has memorized the training data and does not perform well in real-world cases.

Okay, say that I had some training points which look like this:

enter image description here

What if the red curve was the actual 'real-world' relationship. And I found this exact model through a learning algorithm based on those training observations.

Has my model overfit by definition despite it being the 'real-world' relationship? I am assuming yes, but I just want to make sure.



2 Answers 2


I think it may be useful to rephrase the definition of overfitting to something like:

A model that does not generalize well to real-world cases although it fits the training data well.

As for your example:

  • If the real world looks like the red line there is by definition no overfitting.
  • But at the same time, if the black dots are all real-world test data you have, you probably still cannot prove this: in real-world situations, 10 cases are just not enough to prove that a function of the shown complexity was successfully fit.

  • To give you an idea about one real-world field: in analytical chemistry, a series of 10 concentration steps covering your desired range of analyte concentrations is usually required to show that your method yields linear response.


You should look at your error on the training and on the validation data. If the validation error increases while the training error decreases then a situation of overfitting may have occurred. The best predictive and fitted model would be where the validation error has its global minimum.

  • 1
    $\begingroup$ Agreed. The critical point is that overfitting occurs when performance on your training set gets better at a cost of the model generalizing less well. While in this case it certainly looks like what we see when a model overfits, if the real world curve really does match the one your training produced, then your model generalizes as well as can be. $\endgroup$
    – Pat
    Jun 25, 2014 at 16:26
  • $\begingroup$ What if training error is 0 and validation error is 0, have I still overfit by definition? $\endgroup$
    – user46925
    Jun 25, 2014 at 16:28
  • 2
    $\begingroup$ I don't think so, no. Your validation error has not got worse. Take a look at the initial definition you give: "Overfitting is when we have a model which has memorized the training data AND does not perform well in real-world cases". The 'and' there is crucial. Yes, your model has memorized the training set - but it performs excellently on real world cases. So it has not overfit. $\endgroup$
    – Pat
    Jun 25, 2014 at 16:52
  • $\begingroup$ So I have a model that has a lot of variance but does not overfit. Am I correct? Usually, variance and overfitting go together (as they say). $\endgroup$
    – user46925
    Jun 28, 2014 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy