How do I know my model overfits? We already have multiple questions on overfitting, however, we also observe certain kinds of questions to re-occur regularly that ask how to diagnose overfitting in practice. Let's try giving an authoritative answer for such a question, so it could serve as a reference.
How do I know my model overfits? For example: does perfect training error mean overfitting, or how big does the gap between training and test error need to be to call it overfitting?
 A: Re last question: There is no fixed threshold. "Overfitting" is not a binary concept. Particularly, some people will use the term if the test error is any larger than what should be expected from the training error (a certain difference is almost always to be expected), some others may not call a model overfitted that has a substantially larger test error than the training error, as long as the test error is still better than what they get from alternative models. I'd say both are legitimate, and people should say more precisely what they mean. Another issue is that a test error that is achieved on random parts of the same data may still be optimistic when it comes to predicting proper new data.
A: While this question is prone to soliciting "takes" rather than answers, here's mine:
You don't.
Assuming you have used all available resources (i.e. data), regardless of the methodology applied, you don't know if your model is overfit or not. Conversely, modeling techniques that are prone to bias and overfitting may be accurate after all. During independent review, models built using poor methods are dismissed because those methods have poor operating characteristics, but even the crappy archer hits the occasional bullseye. All this is true inspite of any pre-specified objectives, or knowledge of what "plausible" or "good" models might look like apriori. For instance, before the BRCA gene was identified, we had a somewhat pessimistic view of how accurate breast cancer risk models could be, and nothing could have anticipated how powerful their prognostic value might be based on previously researched risk factors.
