# Related to consistency of MLE: why $E_{\theta_0}\log\frac{f(X,\theta)}{f(X,\theta_0)}<0$?

Assumptions.

1. $$\theta_1\ne\theta_2\Rightarrow F_{\theta_1}\ne F_{\theta_2}$$
2. The set $$(x:f(x,\theta)>0)$$ does not depend on $$\theta$$
3. For a.e. $$x$$, $$f(x,\theta)$$ is a differentiable function of $$\theta$$

Theorem. Under these assumptions, for any $$\delta>0$$ $$P_{\theta_0}\left[\frac{\partial \log L_n}{\partial\theta}\text{ has a solution in }(\theta_0-\delta,\theta_0+\delta)\right]\rightarrow1$$ where, $$L_n=L_n(X,\theta)$$

Proof. Observe $$\log\frac{f(X_i,\theta)}{f(X_i,\theta_0)}$$ are iid and $$\frac{1}{n}[\log L_n(\theta)-\log L_n(\theta_0)]=\frac{1}{n}\sum_{i=1}^n\log\frac{f(X_i,\theta)}{f(X_i,\theta_0)}$$ By Jensen's inequality, for all $$\theta\ne\theta_0$$ $$E_{\theta_0}\left[\log\frac{f(X,\theta)}{f(X,\theta_0)}\right]<\log E_{\theta_0}\left[\frac{f(X,\theta)}{f(X,\theta_0)}\right]$$ By WLLN, $$\frac{1}{n}\sum_{i=1}^n\log\frac{f(X_i,\theta)}{f(X_i,\theta_0)}\overset{p}{\longrightarrow}E_{\theta_0}\log\frac{f(X_1,\theta)}{f(X_1,\theta_0)}<0$$ [Rest of the proof.]

I don't see why the last term should be $$<0$$.

• Apply Jensen's inequality, since $\log(\cdot)$ is concave. Commented Mar 29 at 15:14

Excerpt from this lecture note

Because $$\log(\cdot)$$ is strictly concave,

$$\mathbb{E}_{\theta_0}\left[\log\dfrac{f(X,\theta)}{f(X,\theta_0)}\right] < \log\mathbb{E}_{\theta_0}\left[\dfrac{f(X,\theta)}{f(X,\theta_0)}\right] = 0$$

This last equality is because

$$\mathbb{E}_{\theta_0}\left[\dfrac{f(X,\theta)}{f(X,\theta_0)}\right] = \int_\mathcal{X}\dfrac{f(X,\theta)}{f(X,\theta_0)}\cdot {f(X,\theta_0)}\,\mathrm{d}x = \int_\mathcal{X}f(X,\theta)\,\mathrm{d}x = 1$$

and obviously $$\log(1) = 0$$

• It might be useful to add that Jensen's inequality implies $E_{\theta_0}\left[\log\frac{f(X,\theta)}{f(X,\theta_0)}\right] \leq \log \left\{ E_{\theta_0}\left[\frac{f(X,\theta)}{f(X,\theta_0)}\right] \right\}$, where equality holds iff $f(X,\theta)/f(X,\theta_0)$ is constant almost surely which (since $\theta_0$ is identifiable by assumption) holds iff $\theta = \theta_0$. Therefore, the inequality you give holds for $\theta \neq \theta_0$. Commented Mar 29 at 10:41

What you are up to is to show Cramér consistency of MLE (another being the Wald consistency).

As told in the other answer, this is based on Jensen's inequality.

To break it in a systematic way, define $$Y_i:=\ln\{f(X_i\mid\theta)/f(X_i\mid\theta-\varepsilon)\},$$ where $$\theta$$ is the true parameter.

Then \begin{align}\mathbf E_\theta(Y_i)&=\mathbf E_\theta [-\ln\{f(X_i\mid\theta-\varepsilon)/f(X_i\mid\theta)\}]\\&>-\ln \mathbf E_\theta [\{f(X_i\mid\theta-\varepsilon)/f(X_i\mid\theta)\}]\\&>-\ln\int f(x\mid\theta-\varepsilon)~\mathrm d\mu\\&=0.\end{align}

Now by WLLN, $$\frac{\sum Y_i}{n}\to_{\mathbb P_\theta} \mathbf E_\theta (Y_1)> 0.$$ This implies $$\mathbb P_\theta(\sum Y_i> 0)\to 1.$$

In the same way, you can show $$\mathbb P_\theta(\sum Z_i> 0)\to 1$$ for $$Z_i:=\ln\{f(X_i\mid\theta)/f(X_i\mid\theta+\varepsilon)\}.$$

The rest of the proof is elementary.

## Reference:

$$\rm [I]$$ Large Sample Techniques for Statistics, Jiming Jiang, Springer Science$$+$$Business, $$2010,$$ sec. $$1.4,$$ pp. $$9-10.$$