How to understand $E_0(\log f(x;\theta)-\log f(x;\theta_0)) \leq 0$ but $E_0(f(x;\theta)-f(x;\theta_0)) \geq 0$ in MLE?

Question

This problem is about maximum likelihood estimation. Let $f\left(x;\theta\right)$ be a parametric density function with unknown parameter $\theta$, and denote $\theta_{0}$ the true value. By Jensen's inequality, we have $E_{0}\left\{ \log f\left(x;\theta\right)-\log f\left(x;\theta_{0}\right)\right\} \leq0$, where $E_{0}$ indicates that the expectation is under the true density function $f\left(x;\theta_{0}\right)$. In general, we do not have $E_0\left\{ f\left(x;\theta\right)-f\left(x;\theta_{0}\right)\right\} \leq0$ (c.f. the counter example below). This looks counter-intuitive for me. I thought the true density $f(x;\theta_0)$ should be the maximum in the sense that $E_0\left\{ f\left(x;\theta\right)-f\left(x;\theta_{0}\right)\right\} \leq0$. How to understand this? Why is $\log$ transformation indispensable?

Counter-example of $E_0\left\{ f\left(x;\theta\right)-f\left(x;\theta_{0}\right)\right\} \leq0$:

Consider a binary distribution. Let $\pi_{1}$ and $\pi_{2}=1-\pi_{1}$ be true distribution of $X\in\left\{ 0,1\right\} $, and denote $p_{1}$ and $p_{2}=1-p_{1}$ a generic distritbuion of $X$. Here $\pi_1=\Pr(X=0)$ and $p_1=\Pr(X=0)$. The expectation $E_{0}\left\{ f\left(x;\theta\right)-f\left(x;\theta_{0}\right)\right\} $ herein is \begin{eqnarray*} \pi_{1}\left(p_{1}-\pi_{1}\right)+\pi_{2}\left(p_{2}-\pi_{2}\right) & = & \pi_{1}\left(p_{1}-\pi_{1}\right)+\left(1-\pi_{1}\right)\left(\pi_{1}-p_{1}\right)\\ & = & \left(2\pi_{1}-1\right)\left(p_{1}-\pi_{1}\right). \end{eqnarray*} In order to force $E_{0}\left\{ f\left(x;\theta\right)-f\left(x;\theta_{0}\right)\right\} >0$, we simply need $\pi_{1}>1/2$ and $\pi_{1}<p_{1}\leq1$.

Answer 1

The Kullback-Leibler Information inequality holds for the logarithms of any two densities, irrespective of whether one of them is the "true" density describing some population. Therefore, it is equally true that $$E_1\left\{ \log f\left(x;\theta_0\right)-\log f\left(x;\theta_1\right)\right\} \leq0$$ where I have indexed by $1$ the "other" density. We should view these as likelihood functions of the $\theta$'s treated as random variables, not as densities of $X$. This inequality simply states (in totally unscientific terms) that, when you take the expectation of a concave function of "yourself" w.r.t "yourself" you obtain a value that is greater or equal than when you take the expectation of the same concave function of "yourself" w.r.t to "anybody else". This is a direct mathematical result holding under its specific premises - it is not a mathematical formulation of some intuitive truth.

As for the deeper intuition regarding "true" and "not true" densities: A density describes the allocation of probabilities among the population. Assume we deal with discrete random variables, where the density actually gives the probabilities. So "average density" would mean "average probability". What kind of intuition can be attached to "average probability of the elements of a population/sample"? I don't see any - so to expect that this "average probability" should be greater when we sum-product the true density with itself compared to when we sum-product any other density with the true one, also bears no intuition. Why should in general hold that ($f(x)$ is the "true" density)

$$ \sum_i \left[f(x_i)\right]^2 - \sum_i f(x_i)g(x_i) >?0$$