3
$\begingroup$

I understand that, for example, maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood. It is indeed a simple change, but still an extra step taken (it seems) for the unique purpose of designing a loss function that will be minimized instead of maximized.

I wonder why this has become the standard in Machine Learning?

  • Is there any numerical consideration that favors function minimization instead of maximization?
  • Why has gradient descent become such a universal standard? (I have never seen a Deep Learning paper in which they use gradient ascent to directly maximize the likelihood)

Disclaimer :

I came across many similar questions, but none of which that have been truly answered. People typically just explain how both approaches are equivalent, or explain why we use the logarithm for numerical stability, but without explaining why minimization is favored over maximization. (See those two questions : 1, 2)

$\endgroup$
10

6 Answers 6

23
$\begingroup$

Minimising $f(x)$ is entirely equivalent to maximising $-f(x)$, in every aspect: result, numerical precision, computational complexity... everything.

Historically, the convention might have been established because of the "least squares" in linear regression (but don't take my word for it).

If it were the other way round, you'd be asking why we don't minimise some cost function...

$\endgroup$
7
$\begingroup$

You tagged this question with the tag "Maximum Likelihood". In maximum likelihood estimation you explicitly maximize an objective function (namely the likelihood).

It just so happens that for an observation that we assume to be drawn from a Gaussian random variable, the likelihood function usually takes a nice form after you take a logarithm. Then there is usually a leading negation, encouraging the entrepreneurial optimizer to switch away from maximizing the objective to minimizing the negative of objective, or roughly the "cost".

For discrete maximum likelihood estimation the "cost" also has another meaningful name since it takes the same form as the euclidean distance in the observation space. (Note that this notion of distance is always there whether or not you're doing discrete parameter estimation, but it's a little less obvious than the discrete ML estimate which just boils down to picking the nearest valid point). Since there's no such thing as negative distance there is a seemingly strong preference for minimizing the cost and not maximizing the objective in these cases.

You should feel comfortable swapping back and forth between minimizing a cost and maximizing an objective. There is a real reason that ML and MAP estimates specifically choose to maximize an objective function (pdf's are purely positive, and the highest values are quite interesting spots [the mode]), but the practical realization of an estimator is going to be several mathematical manipulations away from the textbook definition.

$\endgroup$
5
  • $\begingroup$ As a side note, if you are wondering why we call it gradient "descent", as opposed to just "gradient optimization" or the like, I would suggest the original french, where the method is used for zero finding, and Cauchy does seem to refer to the value of the function "indefinitely decreas[ing] ... until it vanishes" with only a passing thought as to it ever going negative. Roughly 10 times there Cauchy refers to things getting small, minimizing, decreasing... $\endgroup$
    – Steve Cox
    Dec 28, 2022 at 17:49
  • $\begingroup$ I don't truly know if that's part of the mathematical etymology of "gradient descent" however and can't find an original usage of the term (which might have also been in french by Hadamard) $\endgroup$
    – Steve Cox
    Dec 28, 2022 at 17:50
  • 2
    $\begingroup$ The phrase "gradient descent" also invites an analogy with a ball rolling down a slope. $\endgroup$ Dec 28, 2022 at 20:00
  • 1
    $\begingroup$ @Tanner-reinstateLGBTpeople yes for some reason the 19th century mathematicians don't prefer the analogy of a helium balloon drifting to the top of one of their domed plaster ceilings $\endgroup$
    – Steve Cox
    Dec 28, 2022 at 20:13
  • $\begingroup$ This Answer was merged from a "duplicate" question with different tags. Some points in the answer may no longer apply. $\endgroup$
    – Steve Cox
    Dec 29, 2022 at 15:28
6
$\begingroup$

It's my understanding that the only reason for this distinction is that in numerical analysis, it's the standard to talk about convex optimization rather than concave optimization, even though they are really the same procedures. For example, if you do a google scholar search for "concave optimization", you get about 300,000 hits, but "convex optimization" gets about 2,000,000.

Because convex optimization is talked about more in the numerical analysis literature, this nomenclature is followed in the machine learning community.

As you state, the differences are trivial, so the reason for the distinction is trivial.

$\endgroup$
3
$\begingroup$

History.

A lot of this connects back to estimation in statistics.

For example Gauss. He wanted to estimate the position of an asteroid that was obscured by the sun. He had the idea to minimize the squared error and got much better predictions than his colleagues. When estimating the position of an asteroid, what would be the "gain"? The error cost, however, is easy to see: how far is the asteroid from the expected position.

$\endgroup$
2
$\begingroup$

I suspect it may also be because quite a lot of optimisation algorithms are developed by people working on Operations Research who I think have historically posed problems in terms of minimisation of losses. We minimise because it what the best software supports.

$\endgroup$
1
  • 2
    $\begingroup$ This is a more pragmatic point that we mostly take for granted or more from legacy pov. But it's important to point this out. +1. $\endgroup$ Dec 29, 2022 at 10:19
1
$\begingroup$

You can just as well maximize a function that is equal to -1 times your cost function. It just happens that it's usually more natural to define a function that increases in value the farther from the optimum we get than the reverse, and for that reason we usually try to minimize a function rather than maximize a function.

As an illustrative example of what I mean, see this table, where in the rightmost column you encounter $\operatorname{argmin}$ much more often than $\operatorname{argmax}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.