3
$\begingroup$

In regression/classification problem, we are often interested in minimizing a cost function with respect to the parameters of the model. In many cases, the cost function is the negative likelihood. To minimize it, it is standard to minimize the log of it. As the log is monotonically increasing, the two functions have the same minimums, so the final result of the minimization are the same. Taking the log has the advantage of reducing numerical problems as it transforms products into sums but is there any other advantages?

Question: If we were not concerned by any numerical instabilities, would gradient descent work somehow better on minimizing the negative log-likelihood than the negative likelihood?

Obviously the gradient steps are different. If $f$ is the negative likelihood and $g = -\log(-f)$ the negative log-likelihood, the two steps would be:

$$\Delta w = - \lambda \frac{df(w)}{dw}$$

$$\Delta w = \lambda \frac{df(w)}{dw}\times\frac{1}{f(w)}$$

$\endgroup$
5
$\begingroup$

Numerical stability is by far the most important reason for using the log-likelihood instead of the likelihood. That reason alone is more than enough to choose the log-likelihood over the likelihood. Another reason that jumps to mind is that if there is an analytical solution then it is often much easier to find with the log-likelihood.

The likelihood function is typically a product of likelihood contributions by each observation. Taking the derivative of that will quickly lead to an unmanageable number of cross-product terms due to the product rule. In principle it is possible, but I don't want to be the person to keep track of all those terms.

The log-likelihood transforms that product of individual contributions to a sum of contributions, which is much more manageable due to the sum rule.

$\endgroup$
  • $\begingroup$ Thanks for your answer and good point for the derivative of the product/sum. But like the numerical instability, the point you raise is more a computational issue. My question is about the optimization efficiency using gradient descent. Taking the log defines a different optimization problem and I was wondering if it may potentially lead to more training steps to converge with a gradient descent algorithm. $\endgroup$ – AdeB Apr 10 '15 at 15:47
  • 1
    $\begingroup$ @AdeB You can't just wish away the numerical issues, though! If you're using a computer, you need to care about float precision! Underflow will happen unless you think carefully about what you're doing from a computational perspective. The number of steps is irrelevant when your answer is incorrect. $\endgroup$ – Reinstate Monica Apr 10 '15 at 15:50
  • 2
    $\begingroup$ @user777 I am aware of these numerical issues but this is not my question. What if you only need 1 step to converge with the likelihood and 10^9 steps to converge with the log of the likelihood... $\endgroup$ – AdeB Apr 10 '15 at 15:55
  • 4
    $\begingroup$ Its also worth noting that there are many cases in which the likelihood function is not concave but the log likelihood is concave (we say that the likelihood is "log concave" in this case.) Convex optimization techniques can be used to maximize the log likelihood in such situations even though they aren't applicable to the likelihood. $\endgroup$ – Brian Borchers Apr 11 '15 at 3:39
  • $\begingroup$ Thanks @BrianBorchers, this is the kind of answer I was expecting. $\endgroup$ – AdeB Apr 11 '15 at 13:13
3
$\begingroup$

Gradient descent works (best) when the hessian, ie matrix of second derivatives is orthonormal (up to scaling factor.. Don't remember term) . In other words the error surface is quadratic and same in all directions. Remember the step size is not function of err surface in gradient descent.. If second derivative is rapidly changing (Eg exponential function) then convergence will be slow... A fixed step size will be too big for regions of high curvature and too small for regions of low curvature. So I would claim that taking the log likelihood makes your optimisation surface closer to constant curvature for many typical distributions (Eg normal).
The way I got to grips with this is by doing a Taylor series expansion and considering the requirements to constantly descend the error surface using gradient descent.. You end up( from memory) with choosing the step size as 1/max eigenvalue of error surface (over optimisation region)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.