Why are stationary points on the likelihood function better objective functions?

Question

Gradient descent will seek to find the nearest point at which the objective function is stationary in the direction of the gradient.

According to wikipedia we might be better off finding stationary points on the likelihood estimation instead. Here is the quote;

...it has been long recognized that requiring even local minimization is too restrictive for some problems of maximum-likelihood estimation.Therefore, contemporary statistical theorists often consider stationary points of the likelihood function(or zeros of its derivative, the score function, and other estimating equations).

Source: Stochastic gradient descent

My understanding of the likelihood function is that it is simply the space of parameter likelihoods based on the input data.

How does optimising to find stationary points on the likelihood function differ from simple gradient descent in practice, and why would it be preferable?

Answer 1

The wikipedia passage is misleading in its vagueness. They way it is written it gives the impression that "finding stationary points of a function" is something conflicting with "minimizing/maximizing the function".

While it may be the case that a function acquires a minimum/maximum at a point where its gradient is not zero, "maximum likelihood theorists" most of the time assume away such situations (they mess with limiting distributional properties), and consider instead interior points, which, well, are exactly the stationary points of the function. They also try to prove that the function is concave (for maximization) or convex (for minimization), so that a degree of second-order sufficient conditions for an extremum is obtained (and not of a saddle-point).

Now, how, in a maximum likelihood iterative algorithm, one searches for the stationary points, may differ, there are many ways to do it. One method is Broyden, Fletcher, Goldfarb and Shanno, another is Newton- Raphson, etc.

In short, it is methodologically non-existent to compare/contrast "gradient descent" with "finding stationary points", because the first is a method to achieve the second, which is a goal.