# How does epoch-wise double descent occur if training error is 0?

In this paper, they talk about the existence of epoch-wise double descent. In Figure 10, you can see that, with a sufficiently large model, the test error keeps decreasing even after the training error has become 0.
In my understanding, SGD should not change any weights once the training error is 0, since gradients would also be 0.

So why does the test error keep decreasing? Is it because of the momentum of the optimizer?

• Do the authors define what they're measuring when they say "error"? If they mean "proportion of misclassified samples," then the meaning is obvious -- you can trivially have a nonzero loss function value (e.g. cross entropy) but 0 error.
– Sycorax
Commented Aug 2, 2022 at 22:36
• Ah yes, that must be it. Thanks a lot! Commented Aug 3, 2022 at 9:31

Here's a concrete example. In a two-class classification problem, all negative examples have predicted probability of being positive of 0.49, and all positive examples have predicted probability of being positive of 0.51. Using the decision rule that classifies as positive if the predicted probability is greater than 0.5, otherwise negative, then the misclassification rate is 0. However, the log-loss is $$-n \log 0.49 - p \log 0.51 > 0$$ where $$n > 0$$ is the number of negatives and $$p > 0$$ is the number of positives.
My understanding of the paper you reference is that the authors (Nakkiran et al.) do mean the training error as calculated by the loss function of the training algorithm. Where they discuss epoch-wise double descent, they don't say the training error reached 0, they say it reached $$\approx 0$$ (although I agree fig 10 makes it look like the training error did drop to 0), which they call the interpolation point. So because the error is not quite 0, there's room for the weights to change as training continues.