This question is about the apparent implicit regularization that is observed when training a linear model using SGD. I describe my understanding in the hope that someone can point out what I'm missing.
In Section 5 of Understanding Deep Learning Requires Rethinking Generalization, we are given the problem of fitting a linear model $$y=Xw,$$ where $y$ is the model output, $X$ is the $n \times d$ data matrix with $n$ observations of $d$-dimensional data points and $w$ are the parameters to be learnt. If we let $d \geq n$ then the system has an infinite number of solutions.
The paper goes on to derive the kernel trick (without an embedding into feature space) in the context of SGD. If we run SGD we get a solution of the form $w=X^T\alpha$ due to the update rule. We also have that $y=Xw$. Combining these two, we have $$XX^T\alpha = K \alpha = y.$$
This part (I think) I understand. I don't understand this part:
Note that this kernel solution has an appealing interpretation in terms of implicit regularization. Simple algebra reveals that it is equivalent to the minimum $\ell_2$-norm solution of $Xw=y$.
How is it known that this solution has the minimum $\ell_2$-norm?