Advantages of taking the logarithm to minimize the likelihood In regression/classification problem, we are often interested in minimizing a cost function with respect to the parameters of the model. In many cases, the cost function is the negative likelihood. To minimize it, it is standard to minimize the log of it. As the log is monotonically increasing, the two functions have the same minimums, so the final result of the minimization are the same. Taking the log has the advantage of reducing numerical problems as it transforms products into sums but is there any other advantages?
Question: If we were not concerned by any numerical instabilities, would gradient descent work somehow better on minimizing the negative log-likelihood than the negative likelihood?
Obviously the gradient steps are different. If $f$ is the negative likelihood and $g = -\log(-f)$ the negative log-likelihood, the two steps would be:
$$\Delta w = - \lambda \frac{df(w)}{dw}$$
$$\Delta w = \lambda \frac{df(w)}{dw}\times\frac{1}{f(w)}$$
 A: Numerical stability is by far the most important reason for using the log-likelihood instead of the likelihood. That reason alone is more than enough to choose the log-likelihood over the likelihood. Another reason that jumps to mind is that if there is an analytical solution then it is often much easier to find with the log-likelihood. 
The likelihood function is typically a product of likelihood contributions by each observation. Taking the derivative of that will quickly lead to an unmanageable number of cross-product terms due to the product rule. In principle it is possible, but I don't want to be the person to keep track of all those terms.
The log-likelihood transforms that product of individual contributions to a sum of contributions, which is much more manageable due to the sum rule.
A: Gradient descent works (best)  when the hessian,  ie matrix of second derivatives is orthonormal (up to scaling factor.. Don't remember term) . In other words the error surface is quadratic and same in all directions. Remember the step size is not function of err surface in gradient descent.. If second derivative is rapidly changing (Eg exponential function) then convergence will be slow... A fixed step size will be too big for regions of high curvature and too small for regions of low curvature. So I would claim that taking the log likelihood makes your optimisation surface closer to constant curvature for many typical distributions (Eg normal).
The way I got to grips with this is by doing a Taylor series expansion and considering the requirements to constantly descend the error surface using gradient descent.. You end up( from memory)  with choosing the step size as 1/max eigenvalue of error surface (over optimisation region) 
