Skip to main content
Fix URLs from "https://https://…" to "https://…". The error was caused by a, now fixed, SE UI issue. See: https://meta.stackoverflow.com/q/376047 for more information.
Source Link

Minimizing the loss function $f(\theta)$ with regularization function $g(\theta)$ can be viewed as minimizing a Lagrange FunctionLagrange Function.

$\mathcal{L}(\theta,\lambda) = f(\theta) - \lambda \cdot g(\theta)$

you can then minimize by taking the gradient

$\nabla_{\theta,\lambda} \mathcal{L}(\theta, \lambda)=0$.

In a Lagrange Function, the optimum occurs when the gradient of the loss function is perpendicular to the regularization function.

$\nabla_{\theta} f = \lambda \nabla_{\theta} g,$

Here is a much better explanation by Dikran Marsupial :What is the connection between regularization and the method of Lagrange multipliers ?What is the connection between regularization and the method of Lagrange multipliers ? Just a note it this is a very general explanation.

Minimizing the loss function $f(\theta)$ with regularization function $g(\theta)$ can be viewed as minimizing a Lagrange Function.

$\mathcal{L}(\theta,\lambda) = f(\theta) - \lambda \cdot g(\theta)$

you can then minimize by taking the gradient

$\nabla_{\theta,\lambda} \mathcal{L}(\theta, \lambda)=0$.

In a Lagrange Function, the optimum occurs when the gradient of the loss function is perpendicular to the regularization function.

$\nabla_{\theta} f = \lambda \nabla_{\theta} g,$

Here is a much better explanation by Dikran Marsupial :What is the connection between regularization and the method of Lagrange multipliers ? Just a note it this is a very general explanation.

Minimizing the loss function $f(\theta)$ with regularization function $g(\theta)$ can be viewed as minimizing a Lagrange Function.

$\mathcal{L}(\theta,\lambda) = f(\theta) - \lambda \cdot g(\theta)$

you can then minimize by taking the gradient

$\nabla_{\theta,\lambda} \mathcal{L}(\theta, \lambda)=0$.

In a Lagrange Function, the optimum occurs when the gradient of the loss function is perpendicular to the regularization function.

$\nabla_{\theta} f = \lambda \nabla_{\theta} g,$

Here is a much better explanation by Dikran Marsupial :What is the connection between regularization and the method of Lagrange multipliers ? Just a note it this is a very general explanation.

Source Link

Minimizing the loss function $f(\theta)$ with regularization function $g(\theta)$ can be viewed as minimizing a Lagrange Function.

$\mathcal{L}(\theta,\lambda) = f(\theta) - \lambda \cdot g(\theta)$

you can then minimize by taking the gradient

$\nabla_{\theta,\lambda} \mathcal{L}(\theta, \lambda)=0$.

In a Lagrange Function, the optimum occurs when the gradient of the loss function is perpendicular to the regularization function.

$\nabla_{\theta} f = \lambda \nabla_{\theta} g,$

Here is a much better explanation by Dikran Marsupial :What is the connection between regularization and the method of Lagrange multipliers ? Just a note it this is a very general explanation.