0
$\begingroup$

I am currently programming a sparse bundle adjustment (SBA) for my own problem definition of coordinate calibration. Applying the LM algorithm (slide 10) for parameter optimization, I try to understand how to adapt the damping factor $\lambda$ while performing gradient descent (GD). The crucial step in GD using LM is $\Theta_{i+1} = \Theta_i - (J'J-\lambda I)^{-1}J'e$, where $\Theta$ are the parameters, $J$ is the Jacobian, $I$ the identity matrix, and $e$ the error.

Compared to a (common) Neuronal Network (NN) issues, I am used to have a learning rate $\alpha$ with the GD formulation of $\Theta_{i+1} = \Theta_i - \alpha \Delta\mathcal{L}$, where $\mathcal{L}$ is the loss definition. $\alpha$ can then be adapted by various techniques.

How should I adapt $\lambda$ at all? I know that it has a reciprocal effect of the GD behaviour with $\lambda < 1$ if you are far away from the solution, and $\lambda >> 1$ if you are close. Further, $\lambda = 0$ being the Newton GD and $\lambda = 1$ locks the parameter optimization. That is very empirical and works if I apply parameter adaption during runtime by hand.

Or is it best to keep $\lambda$ fixed and introduce an additional learning rate $\alpha$ to the LM alg.?

$\endgroup$

1 Answer 1

1
$\begingroup$

It is, as you suspect, better to adjust $\lambda$ at each iteration. A rough rule of thumb (there are better methods) is to look at whether the objective function was actually improved by more than a user-specified threshold at each step; if it was, divide $\lambda$ by 10 (moving you closer to a Newton-like update), otherwise, multiply $\lambda$ by 10 (moving you closer to a gradient descent update). In both cases $\lambda$ is subject to upper and lower bounds.

Better, albeit more complex, updating schemes can be found here:

The Levenberg-Marquardt method for nonlinear least squares curve-fitting problems

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.