6
$\begingroup$

This may be a very misinformed question, but I cant figure out why its not true. Here goes:

According to Wikipedia and this post, the hessian of a likelihood function equals the information matrix, or the covariance matrix of the score functions, ie:

$$I(\theta)_{i,j} = \mathrm{E}_\theta \left[ \left(\partial_i \log f_{X\mid\Theta}(X\mid\theta)\right) \left(\partial_j \log f_{X\mid\Theta}(X\mid\theta)\right)\right] \, , = -E\left[\frac{\partial^{2} \log(f(X|\theta))}{\partial \theta_{i} \partial \theta_{j}}\bigg|\theta\right]$$

If this is true, wouldnt these conclusions hold true:

  1. The Hessian of the likelihood functions is always positive semidefinite (PSD)

  2. The likelihood function is thus always convex (since the 2nd derivative is PSD)

  3. The likelihood function will have no local minima, only global minima!!!

These results seem too good to be true, but I cant seem to understand why they are false.

Thanks!

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ You seem to have overlooked the expectation operators in drawing your conclusions. Here is a simple analogy: when the expectation of a random variable is positive, does that imply the variable cannot have any negative values? Your conclusions are tantamount to assuming that it cannot. $\endgroup$ – whuber♦ Feb 27 '14 at 19:34
  • $\begingroup$ @whuber, what you say is true, but practically speaking, if the information matrix, I, can be calculated in closed form, we can always substitute Hessian=H=E(H)=I into our optimization algorithms and thus get a convex problem. In fact, this is exactly what is done in GLMs to insure they converge! However, to me, even this seems to good to be true... $\endgroup$ – DankMasterDan Feb 27 '14 at 19:44
  • $\begingroup$ Quoting the wikipedia article you linked (emphasis added): "In certain cases, the Fisher Information matrix is the negative of the Hessian of the Shannon entropy." Not the hessian of the likelihood: the hessian of the entropy, and only under certain conditions. $\endgroup$ – David Marx Feb 27 '14 at 20:36
  • $\begingroup$ It's not clear whether you already know this (it's possible to read your question to suggest that you do or that you don't), but there are obvious counterexamples to your three conclusions, such as the likelihood for the Cauchy location-parameter. $\endgroup$ – Glen_b Feb 27 '14 at 21:51
8
$\begingroup$

The Fisher Information is defined as

$${\left(\mathcal{I} \left(\theta \right) \right)}_{i, j} = \operatorname{E} \left[\left. \left(\frac{\partial}{\partial\theta_i} \log f(X;\theta)\right) \left(\frac{\partial}{\partial\theta_j} \log f(X;\theta)\right) \right|\theta\right]$$

(the question in the post you linked to states mistakenly otherwise, and the answer politely corrects it).

Under the following regularity conditions:
1) The support of the random variable involved does not depend on the unknown parameter vector
2) The derivatives of the loglikelihood w.r.t the parameters exist up to 3d order
3) The expected value of the squared 1st derivative is finite

and under the assumption that the specification is correct (i.e. the specified distribution family includes the actual distribution that the random variable follows)
then the Fisher Information equals the (negative of the) inverted Hessian of the loglikelihood for one observation. This equality is called the "Information Matrix Equality" for obvious reasons.

While the three regularity conditions are relatively "mild" (or at least can be checked), the assumption of correct specification is at the heart of the issues of statistical inference, especially with observational data. It simply is too strong a condition to be accepted easily. And this is the reason why it is a major issue to prove that the log-likelihood is concave in the parameters (which leads in many cases to consistency and asymptotic normality irrespective of whether the specification is correct -the quasi-MLE case), and not just assume it by assuming that the Information Matrix Equality holds.

So you were absolutely right in thinking "too good to be true".

On the side, you neglected the presence of the minus sign -so the Hessian of the log-likelihood (for one observation) would be negative-semidefinite, as it should since we seek to maximize it, not minimize it.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks @Alecos! So to clarify what you said, if we assume correct specification then (if the first 3 conditions hold), we will achieve a global minima. However, the issue of whether this 'global minima' indeed corresponds to a true global minima in the parameters is determined by whether our specification was correct/true? $\endgroup$ – DankMasterDan Feb 28 '14 at 4:08
  • $\begingroup$ Also, do you know of any source which discusses this in more detail? $\endgroup$ – DankMasterDan Feb 28 '14 at 4:08
  • $\begingroup$ Global maxima. If you cannot prove independently that the log-likelihood is concave in the parameters, then yes, the "globality" will depend on whether the specification is correct. A usual case to prove log-concavity independently, is where you have a distribution with log-concave density (in the variable), and one shows that the log-concavity property holds also w.r.t the parameters (because, say, they are linearly related to the variable). I will look for some references, but it will take a while. $\endgroup$ – Alecos Papadopoulos Feb 28 '14 at 4:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.