# Why do we minimize the negative likelihood if it is equivalent to maximization of the likelihood?

This question has puzzled me for a long time. I understand the use of 'log' in maximizing the likelihood so I am not asking about 'log'.

My question is, since maximizing log likelihood is equivalent to minimizing "negative log likelihood" (NLL), why did we invent this NLL? Why don't we use the "positive likelihood" all the time? In what circumstances is NLL favored?

I found a little explanation here. https://quantivity.wordpress.com/2011/05/23/why-minimize-negative-log-likelihood/, and it seems to explain the obvious equivalence in depth, but does not solve my confusion.

Any explanation will be appreciated.

• Maximum Log Likelihood is not a loss function but its negative is as explained in the article in the last section. It is a matter of consistency. Suppose that you have a smart learning system trying different loss functions for a given problem. The set of loss functions will contain squared loss, absolute loss, etc. To have a consistent list you will add negative log likelihood to the list of loss functions. Mar 10, 2015 at 8:45

This is an alternative answer: optimizers in statistical packages usually work by minimizing the result of a function. If your function gives the likelihood value first it's more convenient to use logarithm in order to decrease the value returned by likelihood function. Then, since the log likelihood and likelihood function have the same increasing or decreasing trend, you can minimize the negative log likelihood in order to actually perform the maximum likelihood estimate of the function you are testing. See for example the nlminb function in R here

• I would say this even goes beyond optimizers and is rooted in the conventions in optimization theory. It seems like minimization is often considered to be the default optimization. For example, consider the name "convex optimization", which goes along with minimization but could have just as easily been called "concave optimization". Mar 10, 2015 at 15:54
• Likelihoods are often very much less than 1 and then the logarithm acts to increase the numbers dealt with.... Aug 28, 2020 at 18:43

Optimisers typically minimize a function, so we use negative log-likelihood as minimising that is equivalent to maximising the log-likelihood or the likelihood itself.

Just for completeness, I would mention that the logarithm is a monotonic function, so optimising a function is the same as optimising the logarithm of it. Doing the log transform of the likelihood function makes it easier to handle (multiplication becomes sums) and this is also numerically more stable. This is because the magnitude of the likelihoods can be very small. Doing a log transform converts these small numbers to larger negative values which a finite precision machine can handle better.

• As an example, I frequently encounter log likelihoods of order -40,000 in my work. In this regime it's numerically impossible to work with the likelihood itself. Mar 14, 2015 at 7:32

Here minimizing means decrease the distance of two distributions to its lowest: the target Bernoulli distribution and the generated result distribution. We measure the distance of two distributions using Kullback-Leibler divergence(also called relative entropy), and due to the large number theory minimizing KL divergence is amount to minimizing cross entropy(either multiclass cross entropy, see here or binary classification, see here and here).

Thus

maximizing log likelihood is equivalent to minimizing "negative log likelihood"

can be translated to

Maximizing the log likelihood is equivalent to minimizing the distance between two distributions, thus is equivalent to minimizing KL divergence, and then the cross entropy.

I think it has become quite intuitive.

The answer is simpler than you might think. It is the convention that we call the optimization objective function a "cost function" or "loss function" and therefore, we want to minimize them, rather than maximize them, and hence the negative log likelihood is formed, rather than positive likelihood in your word. Technically both are correct though. By the way, if we do want to maximize something, usually we call it "utility function" and hence the goal is to maximize them.

The main reason for using log is to handle very small likelihoods. A 32-bit float can only go down to 2^-126 before it gets rounded to 0. It's not just because optimizers are built to minimize functions, since you can easily minimize -likelihood. If you have a large model computing likelihood of a sequence with hundreds of factors it's easy for likelihood to go below floating point limits. Using log transforms 2^-126 into -126, making it more resistant to underflow.

• Welcome to CV. Did you notice the first line of the question states "I am not asking about 'log' "? That is discussed elsewhere. BTW, it's rare for any software to perform MLE using single-precision floats anymore.
– whuber
Aug 28, 2020 at 19:50
• Single-precision is still quite common. Double precision is not going to help much with underflow problems with large models. It will take you to 2^-1022. Log is still required either way. Aug 28, 2020 at 23:16