# Proving that the average of a log-likelihood ratio involving an ML estimator is positive

Consider a random variable X, with a probability density function $f_\theta(x)$, where $\theta \in \Theta$. Denote by $\hat{\theta}(X)$ the Maximum likelihood estimator of the parameter. We know that for any two parameters, $\theta$ and $\theta'$ we have

$$\mathbb{E}_\theta \bigg[\log \frac{f_\theta{(X)}}{f_{\theta'}(X)}\bigg] = D(f_\theta || f_{\theta'})>0$$ where $E_\theta$ is the expectation when $f_\theta(x)$ is the underlying distribution, and $D(f_\theta || f_{\theta'})$ the KL-divergence between $f_\theta$ and $f_{\theta'}$. I would like to know whether a similar inequality holds when $\theta$ is replaced by $\hat{\theta}$. Thus, my question is if the following inequality holds:

$$\mathbb{E}_\theta \bigg[\log \frac{f_{\hat{\theta}(Y)}{(X)}}{f_{\theta'}(X)}\bigg] >0,$$ where in this expression $Y$ and $X$ are independent and identically distributed under $f_\theta$ (i.e., the ML estimate is evaluated at different data points than the pdf $f_\theta$). Intuitively this result should hold since on average the ML estimator will give an estimate close to $\theta$. However, I am not sure if any other assumptions are needed? I would really appreciate any help or hints towards proving this claim. Thanks!

• Normally the notation "$\mathbb{E}_\theta$" means that $\theta$ is a random variable and you are taking expectation with respect to it. That doesn't appear to be the case here, but that makes your question ambiguous: just what are you taking the expectation with respect to? $X$, $Y$, or $(X,Y)$?
– whuber
Apr 5 '18 at 23:20
• $\theta$ is a parameter that is unknown. So in this case $\mathbb{E}_{\theta}$ indicates that we are fixing the parameter to an arbitrary value $\theta$. In this case, $X$ is distributed according to $f_\theta$ and the same with $Y$. $X$ and $Y$ are assumed to be independent. Apr 5 '18 at 23:31
• We do not evaluate the estimator and the log-likelihood ratio at the same observation to avoid bias. Apr 5 '18 at 23:32
• OK--but please explain which variable(s) you are taking the expectation over! Presumably it's both, so that you are comparing a number to $0$ rather than a random variable to $0,$ but you ought to be clearer about what you mean.
– whuber
Apr 6 '18 at 13:28

The inequality cannot hold since \begin{align} \mathfrak{Q}(\theta') &= \mathbb{E}_\theta \bigg[\log \frac{f_{\hat{\theta}(Y)}{(X)}}{f_{\theta'}(X)}\bigg]\\ &= \int \int \log \frac{f_{\hat{\theta}(y)}{(x)}}{f_{\theta'}(x)} f_{\theta}(x)\text{d}x f_{\theta}(y)\text{d}y\\ &= \mathbb{E}_\theta^Y \bigg[\int \log \frac{f_{\hat{\theta}(Y)}{(x)}}{f_{\theta'}(x)} f_{\theta}(x)\text{d}x\bigg] \end{align} is the expectation of a function of $Y$ that is negative at $\theta=\theta'$ and hence in a neighbourhood of $\theta$. Hence $\mathfrak{Q}(\theta')$ is negative in a neighbourhood of $\theta$. (This is called a continuity argument: by continuity, if $𝔔(θ′)$ is negative at $θ′=θ$, it is also negative for values $θ′$ that are close enough to $θ$.)
For instance, in the $\cal{N}(\theta,1)$ case, $$𝔔(θ′)=\frac{1}{2}\mathbb{E}_\theta\left[(X-\theta')^2-(X-Y)^2 \right]=\frac{1}{2}\left[1+(\theta-\theta')^2-2\right]$$ which is negative for $(\theta-\theta')^2<1$.
Note that, on the opposite, $$\mathbb{E}_\theta \bigg[\log \frac{f_{\hat{\theta}(X)}{(X)}}{f_{\theta'}(X)}\bigg]>0$$ since $$\log f_{\hat{\theta}}(x) = \arg\max_\theta f_{\theta}(x)$$
• Interesting argument. However, I forgot to mention that $\theta \neq \theta'$. Do you think there can be any assumption that can lead to that inequality being true? Unfortunately I can't use $X$ for both $\hat{\theta}$ and $f$ because that would introduce bias. Apr 6 '18 at 21:07