Reference: Data-Dependent Early Stopping Criterion for Deep Learning?

Question

In the context of non-parametric regression, this paper provides an data-dependent rule for optimal early stopping, when learning an unknown function $f^{\star}$ lying in some RKHS. Here, one stops the gradient descent procedure which minimizing the loss-function $$ \frac1{n} \sum_{i=1}^n \left( f^{\star}(x_i) - f(x_i|w) \right) , $$ where $f(\cdot|w)$ also belongs to the same RKHS with reproducing kernel $K$ and therefore by the representor theorem admits the convenient form $$ f(\cdot)=\sum_{i=1}^n w_i K(\cdot,x_i) . $$

Is there an analogous data-dependent early stopping rule for feed-forward neural networks with fixed depth $d>1$ and fixed width $W>1$?

Answer 1

Early Stopping as Nonparametric Variational Inference provides an interpretation of each step of SGD as a distribution transformation. In the framework of VI, the ELBO, consisting of an energy term (the usual loss) and an entropy term (of the variational posterior), is maximized.

Instead of directly maximising the ELBO, as in Bayes by Backprop -- here SGD is performed as usual, and terminated when an unbiased estimate of the ELBO reaches it's peak, providing a principled stopping rule.

One challenge which this paper tackles is the estimation of the entropy term, which can be computed as the entropy of the starting weight distribution, plus the log determinant of $I - \alpha H(\theta)$ for every step of SGD, where $H(\theta)$ is the Hessian of the loss.

The authors also comment on the similarity between the above term and the "shrinkage matrix" $\prod_t (I-\alpha_t H_t) $ from the Raskutti paper you linked -- "[the shrinkage matrix] is just the Jacobian of the entire SGD procedure along a particular path".