Do multiple deep descents exist? To my knowledge, the phenomenon of double deep descent is still not well understood, but several authors have reported what they call:

*

*Model-wise double descent ("double descents" observed as models get bigger) This is framed in the abstract as

The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns. However, in the modern practice, very rich models such as neural networks are trained to exactly fit (i.e., interpolate) the data. Classically, such models would be considered over-fit, and yet they often obtain high accuracy on test data. This apparent contradiction has raised questions about the mathematical foundations of machine learning and their relevance to practitioners.



*Sample non-monotonicity ("double descents" as we add data).


*Epoch-wise double descent ("double descents" observed in longer training times)
There are also studies that suggest that these double descents of empirical risk may be explained (at least for the MSE and cross entropy losses) by the fact that variance specifically is mono-modal.
Has this type of non-monotonic phenomenon been reported or formally studied before for more than two descents?
 A: I found two recent works which seem relevant --
Triple descent and the two kinds of overfitting: Where & why do they appear?
The claim is that there are two (sample-wise) peaks: one when number of inputs N equals the input dimension D, and one when N equals the number of parameters P. For linear models, D=P, so only one peak is observed. For highly non-linear deep models, the N=D peak is not as pronounced (and presumably wasn't previously observed). The authors present theoretical justification and empirical observations of both peaks.

On the Multiple Descent of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels
In the setting of kernel "ridgeless" regression, they claim both theoretically and with empirical evidence of multiple descents as the dimensionality of the input data increases (with fixed N) -- to be precise, that there should be a peak at each integer root $D = \sqrt[\leftroot{2}\uproot{2}k]{N}$ of the sample size.


x-axis: log D, y-axis: variance of the minimum-norm interpolant.
Vertical lines denote theoretically predicted peaks.

