Proving that the average of a log-likelihood ratio involving an ML estimator is positive Consider a random variable X, with a probability density function $f_\theta(x)$, where $\theta \in \Theta$. Denote by $\hat{\theta}(X)$ the Maximum likelihood estimator of the parameter. We know that for any two parameters, $\theta$ and $\theta'$ we have
\begin{equation}
\mathbb{E}_\theta \bigg[\log \frac{f_\theta{(X)}}{f_{\theta'}(X)}\bigg] = D(f_\theta || f_{\theta'})>0
\end{equation}
where $E_\theta$ is the expectation when $f_\theta(x)$ is the underlying distribution, and $D(f_\theta || f_{\theta'})$ the KL-divergence between $f_\theta$ and $f_{\theta'}$. I would like to know whether a similar inequality holds when $\theta$ is replaced by $\hat{\theta}$. Thus, my question is if the following inequality holds:
\begin{equation}
\mathbb{E}_\theta \bigg[\log \frac{f_{\hat{\theta}(Y)}{(X)}}{f_{\theta'}(X)}\bigg] >0,
\end{equation}
where in this expression $Y$ and $X$ are independent and identically distributed under $f_\theta$ (i.e., the ML estimate is evaluated at different data points than the pdf $f_\theta$). Intuitively this result should hold since on average the ML estimator will give an estimate close to $\theta$. However, I am not sure if any other assumptions are needed? I would really appreciate any help or hints towards proving this claim. Thanks!
 A: The inequality cannot hold since
\begin{align}
\mathfrak{Q}(\theta') &= \mathbb{E}_\theta \bigg[\log \frac{f_{\hat{\theta}(Y)}{(X)}}{f_{\theta'}(X)}\bigg]\\ &= \int \int \log \frac{f_{\hat{\theta}(y)}{(x)}}{f_{\theta'}(x)} f_{\theta}(x)\text{d}x f_{\theta}(y)\text{d}y\\
&= \mathbb{E}_\theta^Y \bigg[\int \log \frac{f_{\hat{\theta}(Y)}{(x)}}{f_{\theta'}(x)} f_{\theta}(x)\text{d}x\bigg]
\end{align}
is the expectation of a function of $Y$ that is negative at $\theta=\theta'$ and hence in a neighbourhood of $\theta$. Hence $\mathfrak{Q}(\theta')$ is negative in a neighbourhood of $\theta$. (This is called a continuity argument: by continuity, if $(θ′)$ is negative at $θ′=θ$, it is also negative for values $θ′$ that are close enough to $θ$.)
For instance, in the $\cal{N}(\theta,1)$ case,
$$(θ′)=\frac{1}{2}\mathbb{E}_\theta\left[(X-\theta')^2-(X-Y)^2
\right]=\frac{1}{2}\left[1+(\theta-\theta')^2-2\right]$$
which is negative for $(\theta-\theta')^2<1$.
Note that, on the opposite,
$$\mathbb{E}_\theta \bigg[\log \frac{f_{\hat{\theta}(X)}{(X)}}{f_{\theta'}(X)}\bigg]>0$$
since
$$\log f_{\hat{\theta}}(x) = \arg\max_\theta f_{\theta}(x)$$
