18
$\begingroup$

I'm trying to understand at a deeper level the ubiquity of log-likelihood (and perhaps more generally log-probability) in statistics and probability theory. Log-probabilities show up all over the place: we usually work with the log-likelihood for analysis (e.g. for maximization), the Fisher information is defined in terms of the second derivative of the log-likelihood, entropy is an expected log-probability, Kullback-Liebler divergence involves log-probabilities, the expected diviance is an expected log-likelihood, etc.

Now I appreciate the many practical and convenient reasons. Many common and useful pdfs are from exponential families, which leads to elegantly simplified terms when log-transformed. Sums are easier to work with than products (esp for differentiating). Log-probs have a great floating point advantage over straight probs. Log-transforming a pdf often converts a non-concave function into a concave function. But what is the theoretical reason/justification/motivation for log-probs?

As an example of my perplexity, consider the Fisher information (FI). The usual explanation for intuiting the FI is that the second derivative of the log-likelihood tells us how "peaked" the log-likehood is: a highly peaked log-likelihood means the MLE is well-specified and we are relatively sure of its value, while a nearly flat log-likehood (low curvature) means many different parameter values are nearly as good (in terms of the log-likelihood) as the MLE, so our MLE is more uncertain.

This is all well-and-good, but isn't it more natural to just find the curvature of the likelihood function itself (NOT log-transformed)? At first glance the emphasis on the log-transform seems arbitrary and wrong. Surely we are more interested in the curvature of the actual likelihood function. What was Fisher's motivation for working with the score function and the Hessian of the log-likelihood instead?

Is the answer simply that, in the end, we have nice results from the log-likelihood asymptotically? E.g., Cramer-Rao and normality of the MLE/posterior. Or is there a deeper reason?

$\endgroup$
  • 2
    $\begingroup$ I asked a similar question here $\endgroup$ – hxd1011 Jul 6 '17 at 18:05
13
$\begingroup$

It's really just a convenience for loglikelihood, nothing more.

I mean the convenience of the sums vs. products: $\ln (\prod_i x_i) =\sum_i\ln x_i$, the sums are easier to deal with in many respects, such as differentialtion or integration. It's not a convenience for only exponential families, I'm trying to say.

When you deal with a random sample, the likelihoods are of the form: $\mathrm{L}=\prod_ip_i$, so the loglikelihood would break this product into the sum instead, which is easier to manipulate and analyze. It helps that all we care is the point of the maximum, the value at the maximum is not important, se we can apply any monotonous transformation such as the logarithm.

On the curvature intuition. It's basically the same thing in the end as the second derivative of loglikelihood.

UPDATE: This is what I meant on the curvature. If you have a function $y=f(x)$, then it's curvature would be (see (14) on Wolfram): $$\kappa=\frac{f''(x)}{(1+f'(x)^2)^{3/2}}$$

The second derivative of the log likelihood: $$A=(\ln f(x))''=\frac{f''(x)}{f(x)}-\left(\frac{f'(x)}{f(x)}\right)^2$$

At the point of the maximum, the first derivative is obviously zero, so we get: $$\kappa_{max}=f''(x_{max})=Af(x_{max})$$ Hence, my quip that the curvature of the likelihood and the second derivative of loglikelihood are the same thing, sort of.

On the other hand, if the first derivative of likelihood is small not only at but around the point of maximum, i.e. the likelihood function is flat then we get: $$\kappa\approx f''(x)\approx A f(x)$$ Now the flat likelihood is not a good thing for us, because it makes finding the maximum more difficult numerically, and the maximum likelihood is not that better than other points around it, i.e. the parameter estimation errors are high.

And again, we still have the curvature and second derivative relationship. So why didn't Fisher look at the curvature of the likelihood function? I think it's for the same reason of convenience. It's easier to manipulate the loglikelihood because of sums instead of the product. So, he could study the curvature of the likelihood by analyzing the second derivative of the loglikelihood. Although the equation looks very simple for the curvature $\kappa_{max}=f''(x_{max})$, in actuality you're taking a second derivative of the product, which is messier than the sum of second derivatives.

UPDATE 2:

Here's a demonstration. I draw a (completely made up) likelihood function, its a) curvature and b) the 2nd derivative of its log. On the left side you see the narrow likelihood and on the right side it's wide. You see how at the point of the max likelihood a) and b) converge, as they should. More importantly though, you can study the width (or flatness) of the likelihood function by examining the 2nd derivative of its log-likelihood. As I wrote earlier the latter is technically simpler than the former to analyze.

Not surprisingly deeper 2nd derivative of loglikelihood signals flatter likelihood function around its max, which is not desired for it causes bigger parameter estimation error.

enter image description here

MATLAB code in case you want to reproduce the plots:

f=@(x,a)a.^2./(a.^2+x.^2);
c = @(x,a)(-2*a.^2.*(a.^2-3*x.^2)./(a.^2+x.^2).^3/(4*a.^4.*x.^2/(a.^2+x.^2).^4+1).^(3/2));
ll2d = @(x,a)(2*(x.^2-a.^2)./(a.^2+x.^2).^2);

h = 0.1;
x=-10:h:10;

% narrow peak
figure
subplot(1,2,1)
a = 1;
y = f(x,a);
plot(x,y,'LineWidth',2)
%dy = diff(y)/h;
hold on
%plot(x(2:end),dy)
plot(x,c(x,a),'LineWidth',2)
plot(x,ll2d(x,a),'LineWidth',2)
title 'Narrow Likelihood'
ylim([-2 1])

% wide peak
subplot(1,2,2)
a=2;
y = f(x,a);
plot(x,y,'LineWidth',2)
%dy = diff(y)/h;
hold on
%plot(x(2:end),dy)
plot(x,c(x,a),'LineWidth',2)
plot(x,ll2d(x,a),'LineWidth',2)
title 'Wide Likelihood'
legend('likelihood','curvature','2nd derivative LogL','location','best')
ylim([-2 1])

UPDATE 3:

In the code above I plugged some arbitrary bell shaped function into the curvature equation, then calculated the second derivative of its log. I didn't re-scale anything, the values are straight from equations to show the equivalence that I mentioned earlier.

Here's the very first paper on likelihood that Fisher published while still in the university, "On an Absolute Criterion for Fitting Frequency Curves", Messenger of Mathmatics, 41: 155-160 (1912)

As I was insisting all along he doesn't mention any "deeper" connections of log probabilities to entropy and other fancy subjects, neither does he offer his information criterion yet. He simply puts the equation $\log P'=\sum_1^n\log p$ on p.54 then proceeds to talk about maximizing the probabilities. In my opinion, this shows that he was using the logarithm just as a convenient method of analyzing the joint probabilities themselves. It is especially useful in the continuous curve fitting, for which he gives an obvious formula on p.55: $$\log P=\int_{-\infty}^\infty\log fdx$$ Good luck analyzing this likelihood (or probability $P$ as per Fisher) without the log!

One thing to note when reading the paper he was only starting with maximum likelihood estimation work, and did more work in subsequent 10 years, so even the term MLE wasn't coined yet, as far as I know.

$\endgroup$
  • 5
    $\begingroup$ Your final sentence (about curvature) intimates there truly is something fundamental about the log likelihood and that taking logs is not just a mere "convenience." I believe there's much more going on here than you are letting on. $\endgroup$ – whuber Jul 6 '17 at 17:24
  • 2
    $\begingroup$ Your discussion of curvature does not appear relevant, because it doesn't distinguish an analysis of log likelihood from an analysis of the likelihood itself. This answer seems to come down to "logs are convenient," but there's much more to the issue than that, as other answers are beginning to suggest. $\endgroup$ – whuber Jul 7 '17 at 15:55
  • $\begingroup$ @Aksakal OK, thanks, I think I see this now. The likelihood function is specified up to an arbitrary multiplicative constant. Therefore, the value of the likelihood at the maximum, $ f(x_{max}) $, is also arbitrary. For instance, it is common to use a unit normalized likelihood function, where $ f(x_{max}) = 1 $. In that case, the second derivatives of the likelihood and the log-likelihood are equivalent at the maximum. $\endgroup$ – ratsalad Jul 8 '17 at 14:07
  • $\begingroup$ So using the log-likelihood for the Fisher information apparently serves two practical purposes: (1) log-likelihoods are easier to work with, and (2) it naturally ignores the arbitrary scaling factor. And, it gives the same answer as the 2nd deriv of the straight likelihood. This seems an important point to me, one that was not obvious and that I've never seen stated in any statistics text. Presumably it was known to Fisher. $\endgroup$ – ratsalad Jul 8 '17 at 14:07
  • $\begingroup$ $$f(x_{max})''= (\ln f(x))'' f(x_{max})$$ and if the arbitrary multiplicative constant is taken as $f(x_{max}) = 1$ $$f(x_{max})''= (\ln f(x))''$$ $\endgroup$ – ratsalad Jul 8 '17 at 14:19
5
$\begingroup$

Additional point. Some of the commonly used probability distributions (including the normal distribution, the exponential distribution, the Laplace distribution, just to name a few) are log-concave. This means that their logarithm is concave. This makes maximising the log-probability much easier than maximising the original probability (which is particularly handy in maximum likelihood or maximum a-posteriori methods). To give an example, using Newton's method to maximise a multivariate Gaussian distribution directly may take a large number of steps while maximising a paraboloid (the log of the multivariate Gaussian distribution) takes exactly one step.

$\endgroup$
  • 2
    $\begingroup$ Not so fast. See exercise 7.4 on pp. 393-394 of web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf $\endgroup$ – Mark L. Stone Jul 6 '17 at 20:42
  • $\begingroup$ That's not log-concave. The Gaussian is log-concave wrt to its argument or to the mean parameter, not wrt the variance too. If you also want to determine the scale, you can use a normal-gamma distribution, which is also log-concave (using the precision instead of the variance). $\endgroup$ – Luca Citi Jul 6 '17 at 21:23
  • 2
    $\begingroup$ Exactly this. All the talk about how logs are more convenient is nice, but convexity (or concavity, depending on perspective) is what really distinguishes log-likelihood as the "correct" thing to work with. $\endgroup$ – Meni Rosenfeld Jul 7 '17 at 11:26
  • 2
    $\begingroup$ Note that I already mentioned log-concavity in the OP. But this is still just a "convenience", there is no theoretical justification here for log-concavity, and in any case log-likelihoods are not log-concave in general. $\endgroup$ – ratsalad Jul 7 '17 at 15:37
  • 1
    $\begingroup$ @ratsalad, yes, you are right, it's convenience. I think log-probabilities are an additional way to look at a probability function. I can't say for sure which one is better. If you look at [en.wikipedia.org/wiki/… measures), some effectively work on the log-probability (e.g. KL divergence which is effectively the expected value of the difference of log-probabilities), some on the probability directly (e.g. KS distance). $\endgroup$ – Luca Citi Jul 7 '17 at 17:40
4
$\begingroup$

Log-likelihood's theoretical importance can be seen from (at least) two perspectives: asymptotic likelihood theory and information theory.

The earlier of these (I believe) is the asymptotic theory of log-likelihood. I think information theory got underway well after Fisher set maximum likelihood on its course towards 20th century dominance.

In likelihood theory, a parabolic log-likelihood has a central place in inference. Lucien Le Cam has played an important role in elucidating the importance of the quadratic log-likelihood in asymptotic theory.

When you have a quadratic log-likelihood, not only does the curvature about the MLE tell you qualitatively how precisely you can estimate the parameter, but we also know the error is normally distributed with a variance equal to the reciprocal of the curvature. When the log-likelihood is approximately quadratic, then we say these results hold approximately, or asymptotically.

A second reason is the prominence of the log-likelihood (or log-probability) in information theory, where it is the main quantity used to measure the information content.

There is a variant of Entropy called the Kullback-Liebler Divergence that is minimized by the maximum likelihood estimate. In particular, if the true data distribution is $g$ then the "closest" distribution (as measured by the Kullback-Liebler divergence) to $g$ in parametric family $f(\theta)$ is given by $f(\hat{\theta})$, where $ \hat{\theta}$ is the maximum likelihood estimate.

Finally, log-likelihood is the quantity used in various model selection criteria such as AIC and BIC. Essentially, each of these criteria equates an extra parameter/degree of freedom with some multiple of $\ln \hat{L}$.

So, log likelihood, apart from being a useful numerical transformation, has deep ties to inference and information theory.

$\endgroup$
  • $\begingroup$ Your reference to information theory's use of log-likelihoods is circular. Why do they use log? Probably for the same reason, especially, if you consider that information theory's a relatively newer field compared to statistics. $\endgroup$ – Aksakal Jul 7 '17 at 13:33
  • $\begingroup$ @Aksakal yes and no. Information theory got its foundations partially from statistical mechanics and entropy: en.wikipedia.org/wiki/Entropy. Boltzmann defined the entropy of a system using the log of the number of microstates. Why logs? Because it makes entropy/information additive (as your answer points out)? So what? At a numerical level, linearity/additivity opens up the use of the powerful methods of linear algebra. $\endgroup$ – user145807 Jul 7 '17 at 14:00
  • 1
    $\begingroup$ @Aksakal however, on a more fundamental level additivity turns entropy/information into something like a measure ... akin to mass. If you combine two statistically independent systems, then the entropy of the combined system is the sum of the entropy of each system. Here's a nice explainer: physics.stackexchange.com/questions/240636/… $\endgroup$ – user145807 Jul 7 '17 at 14:14
  • 1
    $\begingroup$ @Bey Thermodynamic statistical entropy actually follows directly from the Boltzmann distribution of microstates and classical macroscopic thermo (the form of stat mech entropy was not a "choice"). The Boltzmann distribution itself is a consequence of two premises: (1) the physical property that energies are specified only up to an arbitrary additive constant and (2) the fundamental stat mech assumption that all microstates with the same energy have the same probability. So, at the deepest level thermo entropy involves log-probs because energy is additive and proportional to log-prob. $\endgroup$ – ratsalad Jul 7 '17 at 16:07
  • 2
    $\begingroup$ @ratsalad thanks for expanding on this...as you can see, getting beyond simple "logs are easier" explanations of log-likelihood can take one pretty far afield. I use log-likelihood for the reasons Aksakal gives...however, your OP asked for something deeper. I gave two examples that show connections to other areas that have influenced statistics and likelihood theory. I think the asymptotic explanations are more direct, but entropy and probability are linked in ways that makes log-probabilities things we are interested in beyond mere numerical convenience. $\endgroup$ – user145807 Jul 7 '17 at 16:25
0
$\begingroup$

TLDR: It is much easier to derive sums than products, because the derivative operator is linear with summation but with product u have to do the product rule. It is linear complexity versus some higher order polynomial complexity

$\endgroup$
  • 3
    $\begingroup$ This is what the question means by "convenient and practical." It is far from the only, or even the main, reason why analysis focuses on the log likelihood. Consider, for instance, what the expression for the Fisher Information would look like in terms of the likelihood rather than the log likelihood. $\endgroup$ – whuber Jul 6 '17 at 18:08
  • $\begingroup$ yeah for sure; I think when he said its "easier" to find it directly, I thought he meant the opposite of this, because it certainly is easier to find it after we apply the log transformation. $\endgroup$ – Charlie Tian Jul 6 '17 at 20:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.