Minimizing the loss function $f(\theta)$ with regularization function $g(\theta)$ can be viewed as minimizing a Lagrange FunctionLagrange Function.
$\mathcal{L}(\theta,\lambda) = f(\theta) - \lambda \cdot g(\theta)$
you can then minimize by taking the gradient
$\nabla_{\theta,\lambda} \mathcal{L}(\theta, \lambda)=0$.
In a Lagrange Function, the optimum occurs when the gradient of the loss function is perpendicular to the regularization function.
$\nabla_{\theta} f = \lambda \nabla_{\theta} g,$
Here is a much better explanation by Dikran Marsupial :What is the connection between regularization and the method of Lagrange multipliers ?What is the connection between regularization and the method of Lagrange multipliers ? Just a note it this is a very general explanation.