I have come across the following statement in the textbook A course on Large Sample Theory by Ferguson - Chapter 17. Strong Consistency of the Maximum Likelihood Estimates.

The likelihood ratio, $L_n(\theta)/ L_n(\theta_0)$ converges to zero exponentially fast, at a rate $\text{exp}\{-nK(\theta_0,\theta)\}$

Here, $L_n$ is the likelihood function, $\theta_0$ is the true parameter of the distribution from which data was sampled, $\theta$ is an estimate of the true parameter and $K$ is the KL-divergence between the true distribution and fitted distribution.

The reasoning for the statement was given as follows:

From Weak Law of Large Numbers, we have \begin{aligned} \frac{1}{n} \log \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} &=\frac{1}{n} \sum_{i} \log \frac{L_{i}(\boldsymbol{\theta})}{L_{i}\left(\boldsymbol{\theta}_0\right)} \\ & \stackrel{\mathrm{P}}{\longrightarrow}-\mathrm{K}\left(\boldsymbol{\theta}_0 ,\boldsymbol{\theta}\right), \\ \text{i.e.}\qquad \frac{1}{n} \log \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} + \mathrm{K}\left(\boldsymbol{\theta}_0 ,\boldsymbol{\theta}\right) &= o_p(1) \end{aligned}

I am trying to understand how the above statement on this exponential convergence follows from this convergence in probability.

Here is my attempt where I proceed from the reverse direction:

I need to prove that

\begin{align} & \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} = o_p(-\text{exp}\{-nK (\theta_0,\theta)\}) \\ \implies & \frac{\frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)}}{\text{exp}\{-nK (\theta_0,\theta)\}} = o_p(1) \\ & \text{and because log is a continuous function}, \qquad \log \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} + nK (\theta_0,\theta) = o_p(1) \\ & \text{in other words, I need to establish} \qquad \frac{1}{n} \log \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} + K (\theta_0,\theta) = o_p(1/n) \end{align}

As you can see, I need a convergence order of $o_p(1/n)$, however, from WLLN, I have only $o_p(1)$. How do I proceed from here? Also, in general, is this approach correct? Are there any other ways to prove the same?

  • $\begingroup$ @kjetilbhalvorsen: Appreciate your suggestion to add the self-study tag; added it now. Also, I did provide my attempt and indicated where I am stuck in my original post. Let me know if something was not clear. Thanks. $\endgroup$ May 31, 2021 at 3:42

1 Answer 1


By the Law of Large Numbers $$\frac{1}{n}\log\frac{L_n(\theta)}{L_n(\theta_0)} = -K_n(\theta_0,\theta)+o_p(1)$$ so $$\log\frac{L_n(\theta)}{L_n(\theta_0)} = -nK_n(\theta_0,\theta)+o_p(n)$$

And that's all we need: the term that's proportional to $n$ dominates the $o_p(n)$ term.

If you want the epsilontics: write $r_n$ for the $o_p(n)$ term. For every finite $M$ and $\epsilon>0$ there is $N$ such that for $n>N$ $$P(r_n<Mn)>1-\epsilon.$$ Choose $M=K_n(\theta_0,\theta)/2$, then for every $\epsilon>0$ there is $N$ such that for $n>N$ $$P\left(-nK_n(\theta_0,\theta)+r_n< -nK_n(\theta_0,\theta)/2\right)>1-\epsilon$$ so $$P\left( \frac{L_n(\theta)}{L_n(\theta_0)}<\exp\left(-nK_n(\theta_0,\theta)/2 \right) \right)>1-\epsilon$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.