Is it the case that the log-likelihood *always* has negative curvature? Why?

Question

The Fisher information is defined in two equivalent ways: as the variance of the slope of $\ell(x)$, and as the negative of the expected curvature of $\ell(x)$. Since the former is always positive, this would imply that the curvature of the log-likelihood function is everywhere negative. This seems plausible to me, since every distribution that I have seen has a log-likelihood function with negative curvature, but I don't see why this must be the case.

Answer 1

Your conclusion doesn't follow: if the expected value of the curvature of the log-likelihood is negative, it is not necessarily everywhere negative. It just needs to be, on average, more negative than positive. Think of a bimodal distribution: there is indeed a region in between the modes with positively curved log-likelihood, so your claim cannot be true.

Note the link with maximum likelihood estimation for intuition: in the neighborhood of the MLE, you may expect the curvature to be negative because you are at a maximum (although it is not necessarily, like if the maximum occurs on the boundary, for example). If the curvature is negative in the most likely regions, then the average should tend to be negative, intuitively. In fact, it must always be, under the regularity conditions that allow you to use the equivalency with the "variance of the slope" definition, as you point out.

Answer 2

For some classes of likelihood functions, one can prove that the likelihood is log-concave, i.e. that the log-likelihood has second derivatives $\leq 0$ everywhere, which makes life much easier (e.g. you can often prove the existence of unique global maxima, use specialized optimization methods ...) For example,

• this CV question shows that the exponential-family likelihood with the canonical link function is log-concave
• this paper "Concavity of the Log Likelihood" Pratt 1981, JASA proves log-concavity for a class of models with ordinal responses.

There are certainly counterexamples as well (likelihoods that are provably non-log-concave). For example, any log-likelihood that is bi- or multimodal is non-log-concave ... e.g.

• "A note on bimodality in the log-likelihood function for penalized spline mixed models", Welham and Thompson 2009 CS&DA
• "Flat and Multimodal Likelihoods and Model Lack of Fit in Curved Exponential Families", Sundberg 2010 Scand J Stat
• "Problems with Likelihood Estimation of Covariance Functions of Spatial Gaussian Processes", Warnes and Ripley 1987 Biometrika