14
$\begingroup$

There seems to be conflicting advice out there about how to handle comparing train vs test error, particularly when there is a gap between the two. There seem to be two schools of thought that to me, seem to conflict. I'm looking to understand how to reconcile the two (or understand what I'm missing here).

Thought #1: A gap between train and test set performance alone does not indicate overfitting

First, (also discussed here: How can training and testing error comparisons be indicative of overfitting?), the idea that a difference between train and test set alone cannot indicate overfitting. This agrees with my practical experience with, for example, ensemble tree methods, where even after cross-validation based hyper parameter tuning, the gap between train and test error can remain somewhat large. But (irrespective of model type) as long as you're validation error isn't going back up, you're good. At least, that's the thinking.

Thought #2: When you see a gap between train and test performance: Do things that would combat overfitting

However, then there's advice that you do see, from very good sources that suggest that a gap between train and test error is indicative of overfitting. Here's an example: The "Nuts and Bolts of Deep Learning" talk by Andrew Ng (a fantastic talk) https://www.youtube.com/watch?v=F1ka6a13S9I where at around time stamp 48:00 he draws a flow chart that says "if your train set error is low and your train-dev set error is high, you should add regularization, get more data, or change the model architecture"... which are all actions you might take to combat overfitting.

Which brings me to... : Am I missing something here? Is this a model specific rule of thumb (generally simpler models do appear to have less gap between train and test)? Or are there simply two different schools of thought?

$\endgroup$
5
$\begingroup$

I do not think this is conflicting advice. What we are really interested in is good out-of-sample performance, not in reducing the gap between training and test set performance. If the test set performance is representative of out-of-sample performance (i.e. the test set is large enough, uncontaminated and is a representative sample of the data our model will be applied to), then as long as we get good performance on the test set we are not overfitting, regardless of the gap.

Often, however, if there is a large gap, it may indicate that we could get better test set performance with more regularization/introducing more bias to the model. But that does not mean that a smaller gap means a better model; it's just that if we have a small or no gap between training and test set performance, we know we are definitely not overfitting so adding regularization/introducing more bias to the model will not help.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Interesting point. To summarize "no gap between train and test" means definitely no overfitting, but "some gap between train and test" may or may not mean overfitting. If we go by that logic, the flowchart in Andrew Ng's talk seems a little misleading: it's not as slam dunk as the slide suggests i.e. if you have a gap, you could try regularization or getting more data, but it may not help. Would you agree? $\endgroup$ – ednaMode Jul 29 '17 at 22:54
  • 1
    $\begingroup$ From my experience, yes, I would agree. $\endgroup$ – rinspy Jul 31 '17 at 7:57
  • $\begingroup$ "To summarize no gap between train and test means definitely no overfitting" may not necessarily hold. When you have infinite amount of data, you will get zero gap between train and test even if the model overfit. So I think for that statement to be valid, you need some more assumptions. $\endgroup$ – LKS Apr 13 '18 at 21:35
  • $\begingroup$ @LKS I'm not sure what you mean by overfitting here. Out-of-sample performance will always be less than or equal to in-sample performance, assuming that the distribution of data stays constant. So zero gap is our best case scenario. How can there be overfitting with zero gap? $\endgroup$ – rinspy Apr 16 '18 at 8:35
  • $\begingroup$ @rinspy if you define overfitting as the numerical gap between training and testing data, then your statement is correct. But I would like to reason more about what the model is doing. For example, we have a polynomial function of degree 3 and the result will include a small Gaussian noise. If we have finite samples and use a degree 5 polynomial to fit, there is a large between in and out of sample (prediction) performance. But if we can draw almost infinite samples, a model that purely memorizes will have zero in/out of sample error. $\endgroup$ – LKS Apr 16 '18 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.