I'm finding myself making a gut-feeling judgment more and more often on whether the validation of my model is "close enough" to the modeled results from my training data. I don't recall having ever seen a rule of thumb or guiding principle on how to navigate this problem so I'm using 10%; if my trained model has an R^2 of .70 then I would consider my model to be a valid model (lots of other considerations are taken for model validity) if the validation data has an R^2 of >=.63, for example. I'm looking for either official guidance or generally accepted principles.

  • 1
    $\begingroup$ the answer is obviously no, unless you're in a regulated industry, such as farma. in this case you need to ask elders. they'll know which rule or law is applicable $\endgroup$ – Aksakal Apr 5 '18 at 15:32
  • 1
    $\begingroup$ There really should not be such a rule anyway. The difference between training and validation error is not a measure of overfitting. For example, a correctly fit random forest should generally have close to zero training error. $\endgroup$ – Matthew Drury Apr 5 '18 at 19:06
  • $\begingroup$ I appreciate the comments so far but I'm afraid part of my question might have been taken [a lot] too literally. I obviously don't mean a legally binding rule, I'm curious about generally accepted thoughts on the topic. Also, I'm not fitting a random forest model. It's a linear regression model. I don't consider my experience to be all-inclusive by any means but I've never once fit a model with zero error between the training & validation set. Lastly, maybe we shouldn't say validation is a "measure" but it can absolutely be used to identify whether a model is overfit (or is garbage). $\endgroup$ – L. Rouquette Apr 6 '18 at 13:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.