Given that $X_{1},...,X_{n}$ are i.i.d random variables with joint distribution $f(x\mid \theta) $ with 1 dimensional parameter $\theta$, let $\hat\theta$ be the maximum likelihood estimator of $\theta$.

Based on the Wilks theorem,under null hypothesis that $H_{0}: \theta=\theta_{0}$, one has that $$-2log(f(x\mid \theta_{0})/f(x\mid \hat\theta))\rightarrow \chi_{1}^{2} \space \text{ as } \space n\rightarrow\infty.$$

Since we know that $E\chi_{1}^{2}=1$, is there anyway I can find the value of the following integral (expectation of log likelihood ratio) or at least asymptotics

$$-2\int f(x\mid \theta_{0})log(\frac{f(x\mid \theta_{0})}{f(x\mid \hat\theta)})dx=?$$

My guess is that the integral should be around 1. Can anyone please share some ideas or references concerning the above integral?


The integral in question is in fact the Kullback-Leibler divergance between $F(\cdot,θ_0)$ and $F(\cdot,\hat{θ})$. You cannot generally say that it converges to anything (especially considering the fact that you parametric assumption may be wrong). However, for certain distribution families it has good estimates.

You might want to look at Vladimir Spokoiny publications, who has spent quite a lot of time working on this sort of problems. For instance, you might consider useful this article about exponential family distributions (2005).

  • $\begingroup$ If possible, could you share more reference articles ? thank you. I want to know for what certain distribution families it has good estimates. $\endgroup$ – lzstat Feb 23 '16 at 15:31

You cannot say, in general, that the finite sample expectation of the log likelihood ratio will be 1, even though it asymptotically converges in probability to 1. "Around 1" is a reasonable guess, but it's possible to make the finite sample bias arbitrarily large with any number of contrived distributions.

The actual distribution of the log likelihood ratio under the null hypothesis is extremely complex except when considering distributions which are conserved under convolution, like normal or gamma(theta, 1) RVs.

  • $\begingroup$ How could you say "it asymptotically converges in probability to 1"? thx. $\endgroup$ – lzstat Feb 17 '16 at 19:40
  • $\begingroup$ There is a result showing that likelihood ratios converge asymptotically to the Kullback Liebler divergence. Read here. Under the null, the $f$s are the same, so their divergence is 1 (no difference) $\endgroup$ – AdamO Feb 17 '16 at 19:45
  • $\begingroup$ isn't that if $f$s are same and divergence is zero since $\int p log(p/p)=0$? no difference. The literature you mentioned did give the convergence result. But no information about it expectation or value. $\endgroup$ – lzstat Feb 17 '16 at 21:41
  • $\begingroup$ @lzstat yes and no. Think of it like the Pearson Chi-square test of heterogeneity. You expect cell counts to be the same... but there's always a little discrepancy. The point of testing is how much discrepancy is TOO much. Look at integral you wrote: The MLE is on the bottom, null expected on top, so the density ratio is always greater than 1, thus the log is always greater than 0. and the density is also greater than 0, so this value is always positive valued. It takes a $\chi^2_1$ density under the null. $\endgroup$ – AdamO Feb 18 '16 at 18:59
  • $\begingroup$ @lzstat $\hat{\theta}$ would have to be $\theta_0$ exactly, which is like measuring two people who have the EXACT same height down to the picometer. $\endgroup$ – AdamO Feb 18 '16 at 19:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.