# Convergence rate of log likelihood ratio

I have come across the following statement in the textbook A course on Large Sample Theory by Ferguson - Chapter 17. Strong Consistency of the Maximum Likelihood Estimates.

The likelihood ratio, $$L_n(\theta)/ L_n(\theta_0)$$ converges to zero exponentially fast, at a rate $$\text{exp}\{-nK(\theta_0,\theta)\}$$

Here, $$L_n$$ is the likelihood function, $$\theta_0$$ is the true parameter of the distribution from which data was sampled, $$\theta$$ is an estimate of the true parameter and $$K$$ is the KL-divergence between the true distribution and fitted distribution.

The reasoning for the statement was given as follows:

From Weak Law of Large Numbers, we have \begin{aligned} \frac{1}{n} \log \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} &=\frac{1}{n} \sum_{i} \log \frac{L_{i}(\boldsymbol{\theta})}{L_{i}\left(\boldsymbol{\theta}_0\right)} \\ & \stackrel{\mathrm{P}}{\longrightarrow}-\mathrm{K}\left(\boldsymbol{\theta}_0 ,\boldsymbol{\theta}\right), \\ \text{i.e.}\qquad \frac{1}{n} \log \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} + \mathrm{K}\left(\boldsymbol{\theta}_0 ,\boldsymbol{\theta}\right) &= o_p(1) \end{aligned}

I am trying to understand how the above statement on this exponential convergence follows from this convergence in probability.

Here is my attempt where I proceed from the reverse direction:

I need to prove that

\begin{align} & \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} = o_p(-\text{exp}\{-nK (\theta_0,\theta)\}) \\ \implies & \frac{\frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)}}{\text{exp}\{-nK (\theta_0,\theta)\}} = o_p(1) \\ & \text{and because log is a continuous function}, \qquad \log \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} + nK (\theta_0,\theta) = o_p(1) \\ & \text{in other words, I need to establish} \qquad \frac{1}{n} \log \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} + K (\theta_0,\theta) = o_p(1/n) \end{align}

As you can see, I need a convergence order of $$o_p(1/n)$$, however, from WLLN, I have only $$o_p(1)$$. How do I proceed from here? Also, in general, is this approach correct? Are there any other ways to prove the same?

• Please add the self-study tag & read its wiki. Then tell us what you understand thus far, what you've tried & where you're stuck. We'll provide hints to help you get unstuck. Please make these changes as just posting your homework & hoping someone will do it for you is grounds for closing. May 31 at 0:11
• @kjetilbhalvorsen: Appreciate your suggestion to add the self-study tag; added it now. Also, I did provide my attempt and indicated where I am stuck in my original post. Let me know if something was not clear. Thanks. May 31 at 3:42

By the Law of Large Numbers $$\frac{1}{n}\log\frac{L_n(\theta)}{L_n(\theta_0)} = -K_n(\theta_0,\theta)+o_p(1)$$ so $$\log\frac{L_n(\theta)}{L_n(\theta_0)} = -nK_n(\theta_0,\theta)+o_p(n)$$
And that's all we need: the term that's proportional to $$n$$ dominates the $$o_p(n)$$ term.
If you want the epsilontics: write $$r_n$$ for the $$o_p(n)$$ term. For every finite $$M$$ and $$\epsilon>0$$ there is $$N$$ such that for $$n>N$$ $$P(r_n1-\epsilon.$$ Choose $$M=K_n(\theta_0,\theta)/2$$, then for every $$\epsilon>0$$ there is $$N$$ such that for $$n>N$$ $$P\left(-nK_n(\theta_0,\theta)+r_n< -nK_n(\theta_0,\theta)/2\right)>1-\epsilon$$ so $$P\left( \frac{L_n(\theta)}{L_n(\theta_0)}<\exp\left(-nK_n(\theta_0,\theta)/2 \right) \right)>1-\epsilon$$