# Advantages of taking the logarithm to minimize the likelihood

In regression/classification problem, we are often interested in minimizing a cost function with respect to the parameters of the model. In many cases, the cost function is the negative likelihood. To minimize it, it is standard to minimize the log of it. As the log is monotonically increasing, the two functions have the same minimums, so the final result of the minimization are the same. Taking the log has the advantage of reducing numerical problems as it transforms products into sums but is there any other advantages?

Question: If we were not concerned by any numerical instabilities, would gradient descent work somehow better on minimizing the negative log-likelihood than the negative likelihood?

Obviously the gradient steps are different. If $f$ is the negative likelihood and $g = -\log(-f)$ the negative log-likelihood, the two steps would be:

$$\Delta w = - \lambda \frac{df(w)}{dw}$$

$$\Delta w = \lambda \frac{df(w)}{dw}\times\frac{1}{f(w)}$$

• I've taken the liberty of using mathjax formatting for your math. Please check that it still says what you intend. A Mathjax reference can be found here. meta.math.stackexchange.com/questions/5020/…
– Sycorax
Apr 10, 2015 at 13:57

Numerical stability is by far the most important reason for using the log-likelihood instead of the likelihood. That reason alone is more than enough to choose the log-likelihood over the likelihood. Another reason that jumps to mind is that if there is an analytical solution then it is often much easier to find with the log-likelihood.

The likelihood function is typically a product of likelihood contributions by each observation. Taking the derivative of that will quickly lead to an unmanageable number of cross-product terms due to the product rule. In principle it is possible, but I don't want to be the person to keep track of all those terms.

The log-likelihood transforms that product of individual contributions to a sum of contributions, which is much more manageable due to the sum rule.

• Thanks for your answer and good point for the derivative of the product/sum. But like the numerical instability, the point you raise is more a computational issue. My question is about the optimization efficiency using gradient descent. Taking the log defines a different optimization problem and I was wondering if it may potentially lead to more training steps to converge with a gradient descent algorithm.
Apr 10, 2015 at 15:47
• @AdeB You can't just wish away the numerical issues, though! If you're using a computer, you need to care about float precision! Underflow will happen unless you think carefully about what you're doing from a computational perspective. The number of steps is irrelevant when your answer is incorrect.
– Sycorax
Apr 10, 2015 at 15:50
• @user777 I am aware of these numerical issues but this is not my question. What if you only need 1 step to converge with the likelihood and 10^9 steps to converge with the log of the likelihood...