Why do we minimize the negative likelihood if it is equivalent to maximization of the likelihood?

Question

This question has puzzled me for a long time. I understand the use of 'log' in maximizing the likelihood so I am not asking about 'log'.

My question is, since maximizing log likelihood is equivalent to minimizing "negative log likelihood" (NLL), why did we invent this NLL? Why don't we use the "positive likelihood" all the time? In what circumstances is NLL favored?

I found a little explanation here. https://quantivity.wordpress.com/2011/05/23/why-minimize-negative-log-likelihood/, and it seems to explain the obvious equivalence in depth, but does not solve my confusion.

Any explanation will be appreciated.

Answer 1

This is an alternative answer: optimizers in statistical packages usually work by minimizing the result of a function. If your function gives the likelihood value first it's more convenient to use logarithm in order to decrease the value returned by likelihood function. Then, since the log likelihood and likelihood function have the same increasing or decreasing trend, you can minimize the negative log likelihood in order to actually perform the maximum likelihood estimate of the function you are testing. See for example the nlminb function in R here

Answer 2

Optimisers typically minimize a function, so we use negative log-likelihood as minimising that is equivalent to maximising the log-likelihood or the likelihood itself.

Just for completeness, I would mention that the logarithm is a monotonic function, so optimising a function is the same as optimising the logarithm of it. Doing the log transform of the likelihood function makes it easier to handle (multiplication becomes sums) and this is also numerically more stable. This is because the magnitude of the likelihoods can be very small. Doing a log transform converts these small numbers to larger negative values which a finite precision machine can handle better.

Answer 3

Here minimizing means decrease the distance of two distributions to its lowest: the target Bernoulli distribution and the generated result distribution. We measure the distance of two distributions using Kullback-Leibler divergence(also called relative entropy), and due to the large number theory minimizing KL divergence is amount to minimizing cross entropy(either multiclass cross entropy, see here or binary classification, see here and here).

Thus

maximizing log likelihood is equivalent to minimizing "negative log likelihood"

can be translated to

Maximizing the log likelihood is equivalent to minimizing the distance between two distributions, thus is equivalent to minimizing KL divergence, and then the cross entropy.

I think it has become quite intuitive.

Answer 4

The answer is simpler than you might think. It is the convention that we call the optimization objective function a "cost function" or "loss function" and therefore, we want to minimize them, rather than maximize them, and hence the negative log likelihood is formed, rather than positive likelihood in your word. Technically both are correct though. By the way, if we do want to maximize something, usually we call it "utility function" and hence the goal is to maximize them.

Answer 5

The main reason for using log is to handle very small likelihoods. A 32-bit float can only go down to 2^-126 before it gets rounded to 0. It's not just because optimizers are built to minimize functions, since you can easily minimize -likelihood. If you have a large model computing likelihood of a sequence with hundreds of factors it's easy for likelihood to go below floating point limits. Using log transforms 2^-126 into -126, making it more resistant to underflow.