So I’ve seen the code of an R package where a two dimensional optimisation (actually MLE, finding the minimum of the negative log likelihood) is performed with the optim function and also two optimise functions to find the initial values for that optim function. That is, let $F(x,y)$ be the function whose min to be sought, first fixed $y=y_0$ (some constant) and find $x$ with optimise. Then $x$ is fixed at this optimised value and find $y$ with optimise. If, on the top of these two optimise along the $x$ and $y$ axis, can the initial values of optim be further improved by optimising along, say, diagonally with the use of directional derivative?

  • $\begingroup$ It depends on the context and assumptions. Your description sounds like repeated conditioning on variables, which is used in certain MLE settings. It would be justified when the problem is strictly locally convex. In that case, there's no possible improvement to be made by searching in any other directions, because the package's method will wind up at a local minimum (which will be global when the problem is globally convex). It is easy to construct non-convex optimization problems where searching along coordinate directions does not yield local minima. $\endgroup$ – whuber Mar 16 '19 at 16:45

Yes, steepest gradient descent is a popular method that improve its solution by moving along the steepest descent direction.

Also, you can for instance be creative and let it move along other directions and solve a small optimization problem locally. However, unless the direction that you pick has an advantage (For example, some eigenvector direction or conjugate graidient direction), I don't see the benefits of picking a random direction.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.