12
$\begingroup$

I stumbled upon the following paper Reconciling modern machine learning practice and the bias-variance trade-off and do not completely understand how they justify the double descent risk curve (see below), desribed in their paper.

enter image description here

In the introduction they say:

By considering larger function classes, which contain more candidate predictors compatible with the data, we are able to find interpolating functions that have smaller norm and are thus "simpler". Thus increasing function class capacity improves performance of classifiers.

From this I can understand why the test risk decreases as a function of the function class capacity.

What I don't understand then with this justification, however, is why the test risk increases up to the interpolation point and then decreases again. And why is it exactly at the interpolation point that the number of data points $n$ is equal to the function parameter $N$?

I would be happy if someone could help me out here.

$\endgroup$
2
  • 2
    $\begingroup$ Other posts mentioning this paper, but no duplicates. $\endgroup$ Commented Jun 23, 2021 at 12:07
  • 1
    $\begingroup$ This phenomenon is closely related to how deep learning models do not overfit the training set but achieve almost zero loss, i.e., interpolation counterintuitively against the well known statistical learning theory and of course classical bias-variance trade-off is not resolved. See recent exposition from Orial Vinyal's team Understanding Deep Learning (Still) Requires Rethinking Generalization. >From this I can understand why the test risk decreases as a function of th $\endgroup$ Commented Jun 23, 2021 at 20:54

1 Answer 1

13
$\begingroup$

The main point about Belkin's Double Descent is that, at the interpolation threshold, i.e. the least model capacity where you fit training data exactly, the number of solutions is very constrained. The model has to "stretch" to reach the interpolation threshold with a limited capacity.

When you increase capacity further than that, the space of interpolating solutions opens-up, actually allowing optimization to reach lower-norm interpolating solutions. These tend to generalize better, and that's why you get the second descent on test data.

$\endgroup$
16
  • 4
    $\begingroup$ What is worth highlighting is that this is just a guessxplanation, a hypothesis that cannot really be proven or disproven. This is how the authors try to explain the phenomenon. $\endgroup$
    – Tim
    Commented Jun 23, 2021 at 12:13
  • 1
    $\begingroup$ @Tim It's a formalized hypothesis. They gave multiple experiments corroborating the finding. $\endgroup$
    – Firebug
    Commented Jun 23, 2021 at 12:16
  • $\begingroup$ @Firebug Thank you for answering. However, your answer raises some more questions. Do you mean by "number of solutions" predictors that can fit the training data exactly? If so, why is the number of solutions exactly at this point so much constrained? And why is it, that at this point the training data can be fitted exactly? Why is it, that beyond the interpolation threshold, lower-norm interpolating solutions can be found? $\endgroup$
    – Gilfoyle
    Commented Jun 23, 2021 at 18:39
  • 1
    $\begingroup$ @Samuel "why is it, that at this point the training data can be fitted exactly?" because that's the definition of the interpolation threshold: the point with least capacity where training data can be interpolated. At higher capacity, network parameters have more leeway in how they achieve this interpolation, leading to lower norm solutions. $\endgroup$
    – Firebug
    Commented Jun 24, 2021 at 10:58
  • 1
    $\begingroup$ @adosar That's roughly how I think about it. If we think about (just for analogies sake) interpolating polynomials, they tend to have bad edge effects in between points when having exactly the number of parameters to interpolate a given set of points. That sort of behavior is mitigated by Chebyshev polynomials, but not totally eliminated. On the other hand, spline interpolation (polyharmonic splines) doesn't tend to have these issues. $\endgroup$
    – ttbek
    Commented Jul 26, 2023 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.