7
$\begingroup$

The Fisher information is defined in two equivalent ways: as the variance of the slope of $\ell(x)$, and as the negative of the expected curvature of $\ell(x)$. Since the former is always positive, this would imply that the curvature of the log-liklihood function is everywhere negative. This seems plausible to me, since every distribution that I have seen has a log-likelihood function with negative curvature, but I don't see why this must be the case.

$\endgroup$
15
$\begingroup$

Your conclusion doesn't follow: if the expected value of the curvature of the log-likelihood is negative, it is not necessarily everywhere negative. It just needs to be, on average, more negative than positive. Think of a bimodal distribution: there is indeed a region in between the modes with positively curved log-likelihood, so your claim cannot be true.

Note the link with maximum likelihood estimation for intuition: in the neighborhood of the MLE, you may expect the curvature to be negative because you are at a maximum (although it is not necessarily, like if the maximum occurs on the boundary, for example). If the curvature is negative in the most likely regions, then the average should tend to be negative, intuitively. In fact, it must always be, under the regularity conditions that allow you to use the equivalency with the "variance of the slope" definition, as you point out.

$\endgroup$
11
$\begingroup$

For some classes of likelihood functions, one can prove that the likelihood is log-concave, i.e. that the log-likelihood has second derivatives $<=0$ everywhere, which makes life much easier (e.g. you can often prove the existence of unique global maxima, use specialized optimization methods ...) For example,

  • this CV question shows that the exponential-family likelihood with the canonical link function is log-concave
  • this paper "Concavity of the Log Likelihood" Pratt 1981, JASA proves log-concavity for a class of models with ordinal responses.

There are certainly counterexamples as well (likelihoods that are provably non-log-concave). For example, any log-likelihood that is bi- or multimodal is non-log-concave ... e.g.

$\endgroup$
  • 2
    $\begingroup$ interesting examples! $\endgroup$ – user795305 Mar 14 '17 at 22:23
  • 2
    $\begingroup$ Thank you for adding counterexamples of likelihoods which are not log-concave. $\endgroup$ – Chris Haug Mar 15 '17 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.