2
$\begingroup$

I have a question on second derivative test for most "modern" machine learning algorithms. I learned that in calculus but never seen it in real applications. Most machine learning algorithms optimizations seems to be trying to get the parameters that make derivative to $0$, without further tests to see if it is a max / min or saddle point. Why? Is that because many objective function is not twice differentiable?

I know saddle point is a problem in neural network optimization, but even for simple case, say ridge regression / logistic regression, we also do not do the second derivative test, why? Is that because we know the objective function is convex?

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ In MLP optimization, SGD-like techniques are used. You move down the gradient of the loss, so you explicitly know you won't get stuck in local maxima, but you can get stuck in saddle-points (less likely with momentum and adaptive techniques so common these days) and local minima. $\endgroup$
    – Firebug
    Commented May 3, 2018 at 14:03
  • 3
    $\begingroup$ Logistic regression is a convex optimization problem (mathgotchas.blogspot.com/2011/10/…). Ridge regression is also convex or can be solved as a system of equations. $\endgroup$
    – khol
    Commented May 3, 2018 at 14:05
  • $\begingroup$ @Firebug thanks for the comment. Do you mean that in MLP, many cases we even cannot get converge / gradient 0, so no need to perform further test? $\endgroup$
    – Haitao Du
    Commented May 3, 2018 at 14:05
  • $\begingroup$ @khol I know it can be solved using direct or iterative algorithm (logistic regression needs iterative algorithm), but why no further test? because it is convex? BTW, thanks for the link. $\endgroup$
    – Haitao Du
    Commented May 3, 2018 at 14:06
  • $\begingroup$ Related question "Is it important to have Hessian positive definite for trust region method optimization?". $\endgroup$
    – Richard Hardy
    Commented May 3, 2018 at 14:37

0

Reset to default