In this paper, they talk about the existence of epoch-wise double descent. In Figure 10, you can see that, with a sufficiently large model, the test error keeps decreasing even after the training error has become 0.
In my understanding, SGD should not change any weights once the training error is 0, since gradients would also be 0.
So why does the test error keep decreasing? Is it because of the momentum of the optimizer?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ Do the authors define what they're measuring when they say "error"? If they mean "proportion of misclassified samples," then the meaning is obvious -- you can trivially have a nonzero loss function value (e.g. cross entropy) but 0 error. $\endgroup$– Sycorax ♦Commented Aug 2, 2022 at 22:36
-
$\begingroup$ Ah yes, that must be it. Thanks a lot! $\endgroup$– user2611844Commented Aug 3, 2022 at 9:31
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
I think the answer turns in what the authors mean by "error." If they mean "proportion of misclassified samples," then the meaning is obvious -- you can trivially have a nonzero loss function value (e.g. cross entropy) but 0 error.
Here's a concrete example. In a two-class classification problem, all negative examples have predicted probability of being positive of 0.49, and all positive examples have predicted probability of being positive of 0.51. Using the decision rule that classifies as positive if the predicted probability is greater than 0.5, otherwise negative, then the misclassification rate is 0. However, the log-loss is $-n \log 0.49 - p \log 0.51 > 0$ where $n > 0$ is the number of negatives and $p > 0$ is the number of positives.
$\begingroup$
$\endgroup$
My understanding of the paper you reference is that the authors (Nakkiran et al.) do mean the training error as calculated by the loss function of the training algorithm. Where they discuss epoch-wise double descent, they don't say the training error reached 0, they say it reached $\approx 0$ (although I agree fig 10 makes it look like the training error did drop to 0), which they call the interpolation point. So because the error is not quite 0, there's room for the weights to change as training continues.
As to why this happens - in the original "double descent" paper Belkin et al. suggest that the double descent observed as model size is increased happens because as the amount of over-parameterisation increases, the model learns closer to the "smallest norm function" - in other words, the over-parameterisation acts as a type of regularisation. Nakkiran et al. are suggesting a similar mechanism happens on some large models if training continues past the interpolation point.
