48
$\begingroup$

This question has puzzled me for a long time. I understand the use of 'log' in maximizing the likelihood so I am not asking about 'log'.

My question is, since maximizing log likelihood is equivalent to minimizing "negative log likelihood" (NLL), why did we invent this NLL? Why don't we use the "positive likelihood" all the time? In what circumstances is NLL favored?

I found a little explanation here. https://quantivity.wordpress.com/2011/05/23/why-minimize-negative-log-likelihood/, and it seems to explain the obvious equivalence in depth, but does not solve my confusion.

Any explanation will be appreciated.

$\endgroup$
  • 3
    $\begingroup$ Maximum Log Likelihood is not a loss function but its negative is as explained in the article in the last section. It is a matter of consistency. Suppose that you have a smart learning system trying different loss functions for a given problem. The set of loss functions will contain squared loss, absolute loss, etc. To have a consistent list you will add negative log likelihood to the list of loss functions. $\endgroup$ – Cagdas Ozgenc Mar 10 '15 at 8:45
42
$\begingroup$

This is an alternative answer: optimizers in statistical packages usually work by minimizing the result of a function. If your function gives the likelihood value first it's more convenient to use logarithm in order to decrease the value returned by likelihood function. Then, since the log likelihood and likelihood function have the same increasing or decreasing trend, you can minimize the negative log likelihood in order to actually perform the maximum likelihood estimate of the function you are testing. See for example the nlminb function in R here

$\endgroup$
  • 10
    $\begingroup$ I would say this even goes beyond optimizers and is rooted in the conventions in optimization theory. It seems like minimization is often considered to be the default optimization. For example, consider the name "convex optimization", which goes along with minimization but could have just as easily been called "concave optimization". $\endgroup$ – Bitwise Mar 10 '15 at 15:54
48
$\begingroup$

Optimisers typically minimize a function, so we use negative log-likelihood as minimising that is equivalent to maximising the log-likelihood or the likelihood itself.

Just for completeness, I would mention that the logarithm is a monotonic function, so optimising a function is the same as optimising the logarithm of it. Doing the log transform of the likelihood function makes it easier to handle (multiplication becomes sums) and this is also numerically more stable. This is because the magnitude of the likelihoods can be very small. Doing a log transform converts these small numbers to larger negative values which a finite precision machine can handle better.

$\endgroup$
  • 4
    $\begingroup$ As an example, I frequently encounter log likelihoods of order -40,000 in my work. In this regime it's numerically impossible to work with the likelihood itself. $\endgroup$ – Will Vousden Mar 14 '15 at 7:32
3
$\begingroup$

Here minimizing means decrease the distance of two distributions to its lowest: the target Bernoulli distribution and the generated result distribution. We measure the distance of two distributions using Kullback-Leibler divergence(also called relative entropy), and due to the large number theory minimizing KL divergence is amount to minimizing cross entropy(either multiclass cross entropy, see here or binary classification, see here and here).

Thus

maximizing log likelihood is equivalent to minimizing "negative log likelihood"

can be translated to

Maximizing the log likelihood is equivalent to minimizing the distance between two distributions, thus is equivalent to minimizing KL divergence, and then the cross entropy.

I think it has become quite intuitive.

$\endgroup$
1
$\begingroup$

The answer is simpler than you might think. It is the convention that we call the optimization objective function a "cost function" or "loss function" and therefore, we want to minimize them, rather than maximize them, and hence the negative log likelihood is formed, rather than positive likelihood in your word. Technically both are correct though. By the way, if we do want to maximize something, usually we call it "utility function" and hence the goal is to maximize them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.