2
$\begingroup$

Consider a random variable X, with a probability density function $f_\theta(x)$, where $\theta \in \Theta$. Denote by $\hat{\theta}(X)$ the Maximum likelihood estimator of the parameter. We know that for any two parameters, $\theta$ and $\theta'$ we have

\begin{equation} \mathbb{E}_\theta \bigg[\log \frac{f_\theta{(X)}}{f_{\theta'}(X)}\bigg] = D(f_\theta || f_{\theta'})>0 \end{equation} where $E_\theta$ is the expectation when $f_\theta(x)$ is the underlying distribution, and $D(f_\theta || f_{\theta'})$ the KL-divergence between $f_\theta$ and $f_{\theta'}$. I would like to know whether a similar inequality holds when $\theta$ is replaced by $\hat{\theta}$. Thus, my question is if the following inequality holds:

\begin{equation} \mathbb{E}_\theta \bigg[\log \frac{f_{\hat{\theta}(Y)}{(X)}}{f_{\theta'}(X)}\bigg] >0, \end{equation} where in this expression $Y$ and $X$ are independent and identically distributed under $f_\theta$ (i.e., the ML estimate is evaluated at different data points than the pdf $f_\theta$). Intuitively this result should hold since on average the ML estimator will give an estimate close to $\theta$. However, I am not sure if any other assumptions are needed? I would really appreciate any help or hints towards proving this claim. Thanks!

$\endgroup$
4
  • 1
    $\begingroup$ Normally the notation "$\mathbb{E}_\theta$" means that $\theta$ is a random variable and you are taking expectation with respect to it. That doesn't appear to be the case here, but that makes your question ambiguous: just what are you taking the expectation with respect to? $X$, $Y$, or $(X,Y)$? $\endgroup$
    – whuber
    Apr 5 '18 at 23:20
  • $\begingroup$ $\theta$ is a parameter that is unknown. So in this case $\mathbb{E}_{\theta}$ indicates that we are fixing the parameter to an arbitrary value $\theta$. In this case, $X$ is distributed according to $f_\theta$ and the same with $Y$. $X$ and $Y$ are assumed to be independent. $\endgroup$ Apr 5 '18 at 23:31
  • $\begingroup$ We do not evaluate the estimator and the log-likelihood ratio at the same observation to avoid bias. $\endgroup$ Apr 5 '18 at 23:32
  • 1
    $\begingroup$ OK--but please explain which variable(s) you are taking the expectation over! Presumably it's both, so that you are comparing a number to $0$ rather than a random variable to $0,$ but you ought to be clearer about what you mean. $\endgroup$
    – whuber
    Apr 6 '18 at 13:28
1
$\begingroup$

The inequality cannot hold since \begin{align} \mathfrak{Q}(\theta') &= \mathbb{E}_\theta \bigg[\log \frac{f_{\hat{\theta}(Y)}{(X)}}{f_{\theta'}(X)}\bigg]\\ &= \int \int \log \frac{f_{\hat{\theta}(y)}{(x)}}{f_{\theta'}(x)} f_{\theta}(x)\text{d}x f_{\theta}(y)\text{d}y\\ &= \mathbb{E}_\theta^Y \bigg[\int \log \frac{f_{\hat{\theta}(Y)}{(x)}}{f_{\theta'}(x)} f_{\theta}(x)\text{d}x\bigg] \end{align} is the expectation of a function of $Y$ that is negative at $\theta=\theta'$ and hence in a neighbourhood of $\theta$. Hence $\mathfrak{Q}(\theta')$ is negative in a neighbourhood of $\theta$. (This is called a continuity argument: by continuity, if $𝔔(θ′)$ is negative at $θ′=θ$, it is also negative for values $θ′$ that are close enough to $θ$.)

For instance, in the $\cal{N}(\theta,1)$ case, $$𝔔(θ′)=\frac{1}{2}\mathbb{E}_\theta\left[(X-\theta')^2-(X-Y)^2 \right]=\frac{1}{2}\left[1+(\theta-\theta')^2-2\right]$$ which is negative for $(\theta-\theta')^2<1$.

Note that, on the opposite, $$\mathbb{E}_\theta \bigg[\log \frac{f_{\hat{\theta}(X)}{(X)}}{f_{\theta'}(X)}\bigg]>0$$ since $$\log f_{\hat{\theta}}(x) = \arg\max_\theta f_{\theta}(x)$$

$\endgroup$
1
  • $\begingroup$ Interesting argument. However, I forgot to mention that $\theta \neq \theta'$. Do you think there can be any assumption that can lead to that inequality being true? Unfortunately I can't use $X$ for both $\hat{\theta}$ and $f$ because that would introduce bias. $\endgroup$ Apr 6 '18 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.