15
$\begingroup$

To prevent overfitting people people add a regularization term (proportional to the squared sum of the parameters of the model) with a regularization parameter $\lambda$ to the cost function of linear regression. Is this parameter $\lambda$ the same as a lagrange multiplier? So is regularization the same as the method of lagrange multiplier? Or how are these methods connected?

$\endgroup$
13
$\begingroup$

Say we are optimizing a model with parameters $\vec{\theta}$, by minimizing some criterion $f(\vec{\theta})$ subject to a constraint on the magnitude of the parameter vector (for instance to implement a structural risk minimization approach by constructing a nested set of models of increasing complexity), we would need to solve:

$\mathrm{min}_\vec{\theta} f(\vec{\theta}) \quad \mathrm{s.t.} \quad \|\vec{\theta}\|^2 < C$

The Lagrangian for this problem is (caveat: I think, its been a long day... ;-)

$\Lambda(\vec{\theta},\lambda) = f(\vec{\theta}) + \lambda\|\vec{\theta}\|^2 - \lambda C.$

So it can easily be seen that a regularized cost function is closely related to a constrained optimization problem with the regularization parameter $\lambda$ being related to the constant governing the constraint ($C$), and is essentially the Lagrange multiplier.

This illustrates why e.g. ridge regression implements structural risk minimization: Regularization is equivalent to putting a constraint on the magnitude of the weight vector and if $C_1 > C_2$ then every model that can be made while obeying the constraint that

$\|\vec{\theta}\|^2 < C_2$

will also be available under the constraint

$\|\vec{\theta}\|^2 < C_1$.

Hence reducing $\lambda$ generates a sequence of hypothesis spaces of increasing complexity.

$\endgroup$
3
  • $\begingroup$ "The Lagrangian for this problem is", were you correct in the end or not? Just that this came up having searched for regularisation and lagrangian, and i wanted to double check that the long day didn't have an effect. $\endgroup$
    – baxx
    Dec 29 '20 at 0:50
  • 1
    $\begingroup$ @baxx, yes, I'm pretty sure it is right - there is a similar answer on the maths stack exchage math.stackexchange.com/questions/335306/… $\endgroup$ Jan 2 at 10:55
  • 1
    $\begingroup$ Great answer! What confused me most when comparing to regularization, is that 𝜆 is static, i.e. a fixed and chosen hyperparameter. This answer clarifies that when looking at the Lagrangian, 𝜆 is part of the optimization, but C becomes the hyperparameter instead. $\endgroup$
    – Matt
    Aug 19 at 10:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.