1
$\begingroup$

This problem is about maximum likelihood estimation. Let $f\left(x;\theta\right)$ be a parametric density function with unknown parameter $\theta$, and denote $\theta_{0}$ the true value. By Jensen's inequality, we have $E_{0}\left\{ \log f\left(x;\theta\right)-\log f\left(x;\theta_{0}\right)\right\} \leq0$, where $E_{0}$ indicates that the expectation is under the true density function $f\left(x;\theta_{0}\right)$. In general, we do not have $E_0\left\{ f\left(x;\theta\right)-f\left(x;\theta_{0}\right)\right\} \leq0$ (c.f. the counter example below). This looks counter-intuitive for me. I thought the true density $f(x;\theta_0)$ should be the maximum in the sense that $E_0\left\{ f\left(x;\theta\right)-f\left(x;\theta_{0}\right)\right\} \leq0$. How to understand this? Why is $\log$ transformation indispensable?

Counter-example of $E_0\left\{ f\left(x;\theta\right)-f\left(x;\theta_{0}\right)\right\} \leq0$:

Consider a binary distribution. Let $\pi_{1}$ and $\pi_{2}=1-\pi_{1}$ be true distribution of $X\in\left\{ 0,1\right\} $, and denote $p_{1}$ and $p_{2}=1-p_{1}$ a generic distritbuion of $X$. Here $\pi_1=\Pr(X=0)$ and $p_1=\Pr(X=0)$. The expectation $E_{0}\left\{ f\left(x;\theta\right)-f\left(x;\theta_{0}\right)\right\} $ herein is \begin{eqnarray*} \pi_{1}\left(p_{1}-\pi_{1}\right)+\pi_{2}\left(p_{2}-\pi_{2}\right) & = & \pi_{1}\left(p_{1}-\pi_{1}\right)+\left(1-\pi_{1}\right)\left(\pi_{1}-p_{1}\right)\\ & = & \left(2\pi_{1}-1\right)\left(p_{1}-\pi_{1}\right). \end{eqnarray*} In order to force $E_{0}\left\{ f\left(x;\theta\right)-f\left(x;\theta_{0}\right)\right\} >0$, we simply need $\pi_{1}>1/2$ and $\pi_{1}<p_{1}\leq1$.

$\endgroup$
3
  • $\begingroup$ What does MLE have to do with this? The maximum-likelihood estimate of $\theta_0$ is $\hat{\theta}$ where $\hat{\theta}$ is the number for which $f(x;\theta)$, regarded as a function of $\theta$ (with $x$, the data, being fixed) has maximum value. It is the statistician's fondest hope that $\hat{\theta}$ is close to $\theta_0$. So how does $\hat{\theta}$ relate to the question asked here? $\endgroup$ Jul 29, 2013 at 17:29
  • $\begingroup$ @DilipSarwate As an extreme estimator, the probability limit of MLE $\hat{\theta}$ is the solution of the minimization problem $\min_{\theta} E_0(\log f(x;\theta)-\log f(x;\theta_0))$. $\endgroup$
    – semibruin
    Jul 29, 2013 at 17:48
  • $\begingroup$ The conjecture is wrong. I have found a counter-example in discrete case. $\endgroup$
    – semibruin
    Jul 29, 2013 at 17:49

1 Answer 1

1
$\begingroup$

The Kullback-Leibler Information inequality holds for the logarithms of any two densities, irrespective of whether one of them is the "true" density describing some population. Therefore, it is equally true that $$E_1\left\{ \log f\left(x;\theta_0\right)-\log f\left(x;\theta_1\right)\right\} \leq0$$ where I have indexed by $1$ the "other" density. We should view these as likelihood functions of the $\theta$'s treated as random variables, not as densities of $X$. This inequality simply states (in totally unscientific terms) that, when you take the expectation of a concave function of "yourself" w.r.t "yourself" you obtain a value that is greater or equal than when you take the expectation of the same concave function of "yourself" w.r.t to "anybody else". This is a direct mathematical result holding under its specific premises - it is not a mathematical formulation of some intuitive truth.

As for the deeper intuition regarding "true" and "not true" densities: A density describes the allocation of probabilities among the population. Assume we deal with discrete random variables, where the density actually gives the probabilities. So "average density" would mean "average probability". What kind of intuition can be attached to "average probability of the elements of a population/sample"? I don't see any - so to expect that this "average probability" should be greater when we sum-product the true density with itself compared to when we sum-product any other density with the true one, also bears no intuition. Why should in general hold that ($f(x)$ is the "true" density)

$$ \sum_i \left[f(x_i)\right]^2 - \sum_i f(x_i)g(x_i) >?0$$

$\endgroup$
2
  • $\begingroup$ @ Alecos Papadopoulos Thanks for your answer. It clarified a lot. The right thing I shall expect is not $E_0(f(X;\theta_0))\geq E_0(f(X;\theta))$, because as you said $E(f(X;\theta))$ is only the average "height" of the function $f(X;\theta)$. Given sample $X_1,\ldots, X_n$, what I should expect is that $f(X_1,\ldots,X_n;\theta_0) \geq f(X_1,\ldots,X_n;\theta)$ as $n\rightarrow \infty$. When the samples are iid, $f(X_{1},\ldots,X_{n};\theta)=\prod_{i=1}^{n}f\left(X_{i};\theta\right)$. $\endgroup$
    – semibruin
    Aug 12, 2013 at 6:33
  • $\begingroup$ ...As a monotone transformation, we expect that $\sum_{i=1}^{n}\log f\left(X_{i};\theta_{0}\right)\geq\sum_{i=1}^{n}\log f\left(X_{i};\theta\right)$. By the LLN, we then can expect $E_{0}\left(\log f\left(X;\theta_{0}\right)\right)\geq E_{0}\left(\log f\left(X;\theta\right)\right)$. Things make better sense now. $\endgroup$
    – semibruin
    Aug 12, 2013 at 6:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.