1
$\begingroup$

I understand that, for example, maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood. It is indeed a simple change, but still an extra step taken (it seems) for the unique purpose of designing a loss function that will be minimized instead of maximized.

I wonder why this has become the standard in Machine Learning?

  • Is there any numerical consideration that favors function minimization instead of maximization?
  • Why has gradient descent become such a universal standard? (I have never seen a Deep Learning paper in which they use gradient ascent to directly maximize the likelihood)

Disclaimer :

I came across many similar questions, but none of which that have been truly answered. People typically just explain how both approaches are equivalent, or explain why we use the logarithm for numerical stability, but without explaining why minimization is favored over maximization. (See those two questions : 1, 2)

$\endgroup$
  • 4
    $\begingroup$ The distinction between minimization of an objective and maximization of the negative of the objective function is so trivial that nobody is concerned about it. Many authors and software implementations have settled on the minimization convention, that's all. One can scarcely characterize this as an "extra step"--it's no different than having to pay attention to any other convention, such as what order to pass parameters to a function or whether to record lengths in feet or meters. $\endgroup$ – whuber Oct 17 '17 at 19:38
  • 2
    $\begingroup$ I've heard that although they are mathematically identical, minimization is preferred due to the nature of the numerical computations the computer has to do. If so, it may be that someone on, say, Computational Science, Computer Science, or Theoretical Computer Science could provide a good explanation. $\endgroup$ – gung - Reinstate Monica Oct 17 '17 at 20:07
  • $\begingroup$ @gung The only difference in most digital computers between a negative and positive (floating point) value is the setting of the sign bit. There's no difference in precision or in performance of the numerical operations. $\endgroup$ – whuber Oct 17 '17 at 20:28
  • 2
    $\begingroup$ Simply a convention in optimization - see 1, 2, 3 ... ctd $\endgroup$ – Glen_b -Reinstate Monica Oct 17 '17 at 21:40
  • $\begingroup$ ctd... Also see this, which makes it clear that it is effectively arbitrary. $\:$ It makes sense to settle on a single convention, so that people that work on the mathematics of optimization algorithms (and the people who code them) can focus on the essentials instead of constantly converting back and forth between different optimization conventions (these authors minimize; those maximize). Ultimately it also makes it easier on users. $\endgroup$ – Glen_b -Reinstate Monica Oct 17 '17 at 21:40
3
$\begingroup$

It's my understanding that the only reason for this distinction is that in numerical analysis, it's the standard to talk about convex optimization rather than concave optimization, even though they are really the same procedures. For example, if you do a google scholar search for "concave optimization", you get about 300,000 hits, but "convex optimization" gets about 2,000,000.

Because convex optimization is talked about more in the numerical analysis literature, this nomenclature is followed in the machine learning community.

As you state, the differences are trivial, so the reason for the distinction is trivial.

$\endgroup$
1
$\begingroup$

History.

A lot of this connects back to estimation in statistics.

For example Gauss. He wanted to estimate the position of an asteroid that was obscured by the sun. He had the idea to minimize the squared error and got much better predictions than his colleagues. When estimating the position of an asteroid, what would be the "gain"? The error cost, however, is easy to see: how far is the asteroid from the expected position.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.