3
$\begingroup$

I am confused on the definitions of steepest descent.

  • In some literature, such as this and this, steepest descent means using negative gradient direction and exact line search on that direction.
  • But in this note, It seems as well as we are following negative gradient, the method can be called steepest descent.

Which one is correct? Can we call using fixed alpha (without line search) in negative gradient direction steepest descent?

Is the term "steepest descent" loosely defined?

$\endgroup$
  • 3
    $\begingroup$ Gradient based optimization is just any method that uses gradients to optimize a function. Thus, gradient descent, Newton's method and L-BFGS are all just examples of gradient based optimization. This is comparison with gradient free methods, such as bisection method, Nelder Mead, genetic algorithms, etc. $\endgroup$ – Cliff AB Jan 8 '18 at 23:16
  • $\begingroup$ @CliffAB I think the wording in my original question may be confusing. I know what is gradient based optimization, but just want to ask the definition of steepest decent. I revised it. Thanks for the comment. $\endgroup$ – Haitao Du Jan 9 '18 at 15:09
4
$\begingroup$

Steepest descent is a special case of gradient descent where the step length is chosen to minimize the objective function value. Gradient descent refers to any of a class of algorithms that calculate the gradient of the objective function, then move "downhill" in the indicated direction; the step length can be fixed, estimated (e.g., via line search), or ... (see this link for some examples).

Gradient-based optimization is, as Cliff AB points out in comments to the OP, more general still, referring to any method that uses gradients to optimize a function. Note that this does not mean you necessarily move in the direction that would be indicated by the gradient (see, for example, Newton's method.)

$\endgroup$
  • $\begingroup$ +1 for the answer. I think I have a terminology question: if we used fixed step size and negative gradient direction it is "steepest" or not. Different literature seems define it differently. $\endgroup$ – Haitao Du Jan 9 '18 at 15:03
  • $\begingroup$ I revised my question to avoid confusions, if you have time please take a look $\endgroup$ – Haitao Du Jan 9 '18 at 15:08
  • $\begingroup$ It is true that some people, sloppily IMO, define "steepest descent" to be any algorithm that steps in the steepest descent direction regardless of how the step length is determined. However, the actual steepest descent algorithm not only steps in the steepest descent direction but determines step length to minimize the objective function in that direction. See for example math.usm.edu/math/lambers/mat419/lecture10.pdf. But not many people will complain if you use the looser definition, not even me. $\endgroup$ – jbowman Jan 9 '18 at 17:29
0
$\begingroup$

Gradient is a multi-variable generalization of the derivative (at a point). While a derivative can be defined on functions of a single variable, for functions of several variables.

Since descent is negative sloped, and to perform gradient descent, we are minimizing error, then maximum steepness is the most negative slope.

$\endgroup$
  • $\begingroup$ Thanks for the answer. this is exactly what I have the confusion. Suppose we use gradient decent with fixed step size, is that "steepest decent"? Some literature says yes, other says no, because it is not using the "exact line search" $\endgroup$ – Haitao Du Jan 9 '18 at 15:01
  • $\begingroup$ Among other things, steepest descent is the name of an algorithm. See gradient descent methods. I think part of your confusion is due to admixing terms from slightly different algorithms. $\endgroup$ – Carl Jan 9 '18 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.