Proving that the average of a log-likelihood ratio involving an ML estimator is positive

Question

Consider a random variable X, with a probability density function $f_\theta(x)$, where $\theta \in \Theta$. Denote by $\hat{\theta}(X)$ the Maximum likelihood estimator of the parameter. We know that for any two parameters, $\theta$ and $\theta'$ we have

\begin{equation} \mathbb{E}_\theta \bigg[\log \frac{f_\theta{(X)}}{f_{\theta'}(X)}\bigg] = D(f_\theta || f_{\theta'})>0 \end{equation} where $E_\theta$ is the expectation when $f_\theta(x)$ is the underlying distribution, and $D(f_\theta || f_{\theta'})$ the KL-divergence between $f_\theta$ and $f_{\theta'}$. I would like to know whether a similar inequality holds when $\theta$ is replaced by $\hat{\theta}$. Thus, my question is if the following inequality holds:

\begin{equation} \mathbb{E}_\theta \bigg[\log \frac{f_{\hat{\theta}(Y)}{(X)}}{f_{\theta'}(X)}\bigg] >0, \end{equation} where in this expression $Y$ and $X$ are independent and identically distributed under $f_\theta$ (i.e., the ML estimate is evaluated at different data points than the pdf $f_\theta$). Intuitively this result should hold since on average the ML estimator will give an estimate close to $\theta$. However, I am not sure if any other assumptions are needed? I would really appreciate any help or hints towards proving this claim. Thanks!

Answer 1

The inequality cannot hold since \begin{align} \mathfrak{Q}(\theta') &= \mathbb{E}_\theta \bigg[\log \frac{f_{\hat{\theta}(Y)}{(X)}}{f_{\theta'}(X)}\bigg]\\ &= \int \int \log \frac{f_{\hat{\theta}(y)}{(x)}}{f_{\theta'}(x)} f_{\theta}(x)\text{d}x f_{\theta}(y)\text{d}y\\ &= \mathbb{E}_\theta^Y \bigg[\int \log \frac{f_{\hat{\theta}(Y)}{(x)}}{f_{\theta'}(x)} f_{\theta}(x)\text{d}x\bigg] \end{align} is the expectation of a function of $Y$ that is negative at $\theta=\theta'$ and hence in a neighbourhood of $\theta$. Hence $\mathfrak{Q}(\theta')$ is negative in a neighbourhood of $\theta$. (This is called a continuity argument: by continuity, if $𝔔(θ′)$ is negative at $θ′=θ$, it is also negative for values $θ′$ that are close enough to $θ$.)

For instance, in the $\cal{N}(\theta,1)$ case, $$𝔔(θ′)=\frac{1}{2}\mathbb{E}_\theta\left[(X-\theta')^2-(X-Y)^2 \right]=\frac{1}{2}\left[1+(\theta-\theta')^2-2\right]$$ which is negative for $(\theta-\theta')^2<1$.

Note that, on the opposite, $$\mathbb{E}_\theta \bigg[\log \frac{f_{\hat{\theta}(X)}{(X)}}{f_{\theta'}(X)}\bigg]>0$$ since $$\log f_{\hat{\theta}}(x) = \arg\max_\theta f_{\theta}(x)$$