Convergence rate of log likelihood ratio

Question

I have come across the following statement in the textbook A course on Large Sample Theory by Ferguson - Chapter 17. Strong Consistency of the Maximum Likelihood Estimates.

The likelihood ratio, $L_n(\theta)/ L_n(\theta_0)$ converges to zero exponentially fast, at a rate $\text{exp}\{-nK(\theta_0,\theta)\}$

Here, $L_n$ is the likelihood function, $\theta_0$ is the true parameter of the distribution from which data was sampled, $\theta$ is an estimate of the true parameter and $K$ is the KL-divergence between the true distribution and fitted distribution.

The reasoning for the statement was given as follows:

From Weak Law of Large Numbers, we have \begin{aligned} \frac{1}{n} \log \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} &=\frac{1}{n} \sum_{i} \log \frac{L_{i}(\boldsymbol{\theta})}{L_{i}\left(\boldsymbol{\theta}_0\right)} \\ & \stackrel{\mathrm{P}}{\longrightarrow}-\mathrm{K}\left(\boldsymbol{\theta}_0 ,\boldsymbol{\theta}\right), \\ \text{i.e.}\qquad \frac{1}{n} \log \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} + \mathrm{K}\left(\boldsymbol{\theta}_0 ,\boldsymbol{\theta}\right) &= o_p(1) \end{aligned}

I am trying to understand how the above statement on this exponential convergence follows from this convergence in probability.

Here is my attempt where I proceed from the reverse direction:

I need to prove that

\begin{align} & \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} = o_p(-\text{exp}\{-nK (\theta_0,\theta)\}) \\ \implies & \frac{\frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)}}{\text{exp}\{-nK (\theta_0,\theta)\}} = o_p(1) \\ & \text{and because log is a continuous function}, \qquad \log \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} + nK (\theta_0,\theta) = o_p(1) \\ & \text{in other words, I need to establish} \qquad \frac{1}{n} \log \frac{L_n(\boldsymbol{\theta})}{L_n\left(\boldsymbol{\theta}_0 \right)} + K (\theta_0,\theta) = o_p(1/n) \end{align}

As you can see, I need a convergence order of $o_p(1/n)$, however, from WLLN, I have only $o_p(1)$. How do I proceed from here? Also, in general, is this approach correct? Are there any other ways to prove the same?

Answer 1

By the Law of Large Numbers $$\frac{1}{n}\log\frac{L_n(\theta)}{L_n(\theta_0)} = -K_n(\theta_0,\theta)+o_p(1)$$ so $$\log\frac{L_n(\theta)}{L_n(\theta_0)} = -nK_n(\theta_0,\theta)+o_p(n)$$

And that's all we need: the term that's proportional to $n$ dominates the $o_p(n)$ term.

If you want the epsilontics: write $r_n$ for the $o_p(n)$ term. For every finite $M$ and $\epsilon>0$ there is $N$ such that for $n>N$ $$P(r_n<Mn)>1-\epsilon.$$ Choose $M=K_n(\theta_0,\theta)/2$, then for every $\epsilon>0$ there is $N$ such that for $n>N$ $$P\left(-nK_n(\theta_0,\theta)+r_n< -nK_n(\theta_0,\theta)/2\right)>1-\epsilon$$ so $$P\left( \frac{L_n(\theta)}{L_n(\theta_0)}<\exp\left(-nK_n(\theta_0,\theta)/2 \right) \right)>1-\epsilon$$