# Distribution for which the log-likelihood evaluated at the ML estimate is not equal to the expected log-likelihood evaluated at the ML estimate

Suppose that for $t=1,\dots,T$, $x_t$ is an i.i.d. draw from a continuous distribution with p.d.f. $f(x_t;\theta_0)$.

Let $l(\theta;X):=\sum_{t=1}^T{\log{f(x_t;\theta)}}$ be the corresponding log-likelihood.

Let $\hat\theta:=\arg\max_\theta{l(\theta;X)}$ be the corresponding ML estimate.

Finally, define:

$$e:=l(\hat\theta;X)-\left.\left[\mathbb{E}l(\theta;X)\right]\right|_{\theta_0=\hat\theta, \theta=\hat\theta}$$

Now, for fairly standard distributions, $e=0$. E.g. for the normal distribution: $$l(\hat\theta;X)=-\frac{T}{2}\log{2\pi}-T\log\hat\sigma-\frac{T}{2}=\left.\left[\mathbb{E}l(\theta;X)\right]\right|_{\theta_0=\hat\theta, \theta=\hat\theta}.$$

Is this really a general property though? I suppose it is equivalent to the assertion that the log-likelihood evaluated at the ML estimate is only a function of the data via $\hat\theta$. This sounds a bit like $\hat\theta$ being a sufficient statistic, so perhaps it is only a property of distributions from the exponential family. (Side question: is this a theorem?)

My main questions then are the following:

Can you find a distribution $f$ for which $e\ne 0$?

Does one such $f$ exist for which the ML estimator admits a closed form solution?

• I am confused by the "double conditioning" of the expectation. You write $|\theta_0 = \hat \theta$ . This could be translated "if it so happens that the obtained estimate equals the true value exactly". Is this the intended meaning? Also, you then add $|\theta = \hat \theta$. In all? May 10, 2017 at 19:40
• You were right to query this, as what I'd written was not as clear as it should have been. I have edited the post now, and this alternative notation is I think clearer. Note that the double conditioning is needed as $\mathbb{E}[l(\theta,X)]$ is a function of both $\theta$ (explicitly) and $\theta_0$ (as the distribution of $X$ is).
– cfp
May 11, 2017 at 21:28
• Indeed it is clearer now. I think I will update my answer accordingly May 11, 2017 at 21:37
– cfp
May 13, 2017 at 16:38
• I am thinking about your other comment also, below my answer. May 13, 2017 at 17:28

To re-examine the Uniform distribution example given by @jbowman, consider any i.i.d. sample $\{\mathbf x_i;i=1,...,n\}$ from a continuous uniform distribution $U(0,\theta)$. The log-likelihood function is

$$\ell (\theta;\mathbf x_i) = -n\ln \theta + \ln\left (I_{\{ 0\leq x_i \leq \theta;\forall i\}}\right)$$

To consider non-trivially the expected value of the log-likelihood, we have to treat the $x_i$'s as random variables, not as realized values from a specific sample. On the other hand, assuming that $\theta_0 = \hat \theta$, means that $\hat \theta$, although it will be calculated using the $x's$, it should be treated as a constant in the log-likelihood. Then

$$\mathbb{E}\left[\ell(\theta;\mathbf x_i)\middle|\theta_0=\hat\theta, \theta=\hat \theta \right] = -n\mathbb{E}_{\hat \theta}\left[\ln \hat \theta+\ln\left (I_{\{ 0\leq x_i \leq \hat \theta;\forall i\}}\right)\right]$$

But since we calculate this expected value under the assumption that $\hat \theta$ is the true value, it follows that, even though the $x_i$'s are viewed as random variables, we have $I_{\{ 0\leq x_i \leq \hat \theta;\forall i\}}=1$ always. So we end up with

$$\mathbb{E}\left[\ell(\theta;\mathbf x_i)\middle|\theta_0=\hat\theta, \theta=\hat \theta\right]= -n\mathbb{E}_{\hat \theta}[\ln \hat \theta]=-n\ln \hat \theta=\ell (\hat \theta;\mathbf x_i)$$

So we obtain here too the same equality result.

• Having clarified the conditioning in the question, I'm not sure this works actually. $l(\theta,X)=-n\log\theta+log(I_{\hat\theta \le \theta})$, so $\mathbb{E}l(\theta,X)=-n\log\theta+\mathbb{E} log(I_{\hat\theta \le \theta}) = -n\log\theta + 0$ if $\theta_0 \le \theta$, $-\infty$ otherwise. Thus, if $\theta_0=\theta$, the second term drops out, so $\left.\left[\mathbb{E}l(\theta,X)\right]\right|_{\theta_0=\hat\theta,\theta=\hat\theta}=-n\log\hat\theta=l(\hat\theta,X)$.
– cfp
May 11, 2017 at 22:04
• @cfp You are right. May 13, 2017 at 18:59

Consider the uniform distribution over $(0, \theta), \theta = 1$. $\mathbb{E}[l(\theta; X)|\theta = 1] = 0$, for obvious reasons. However, since $\hat{\theta} = \max(x_1, \dots, x_n) < 1$, $l(\hat{\theta};X) = \log(1/\hat{\theta}) > 0$.

Your conjecture that it is only a property of distributions from the exponential family is an interesting one, but I am not aware of any results along those lines (although there are probably > 10,000 people who would be more likely than I to be aware of such results.)

• But $\left. \mathbb{E}[l(\theta;X) \mid \theta_0=\hat\theta] \right|_{\theta = \hat\theta}$ is different from $\mathbb{E}[l(\theta;X) \mid \theta=1]$ which is what you calculated. The former is the expected value of the likelihood (evaluated at $\hat\theta$) in the hypothetical case that $\hat\theta$ was the true parameter. May 10, 2017 at 19:39
• Bridgeburners is correct, but from Alecos Papadopoulos's answer above, it does look like jbowman found a great example.
– cfp
May 11, 2017 at 21:02

I found a simple example that works at last, albeit a slightly unsatisfying one.

Suppose $x_t\sim\mathrm{NIID}(\theta_0,1)$. Then: $$l(\theta;X)=-\frac{T}{2}\log{2\pi}-\frac{1}{2}\sum_{t=1}^T{\left(x_t-\theta\right)^2},$$ so: \begin{align}l(\hat\theta;X) & =-\frac{T}{2}\log{2\pi}-\frac{1}{2}\sum_{t=1}^T{\left(x_t-\frac{1}{T}\sum_{s=1}^T{x_s}\right)^2} \\ & =-\frac{T}{2}\log{2\pi}-\frac{1}{2}\sum_{t=1}^T{x_t^2}+\frac{1}{2T}\sum_{s=1}^T{\sum_{t=1}^T{x_s x_t}}.\end{align} However: $$\mathbb{E}[l(\theta;X)]=-\frac{T}{2}\log{2\pi}-\frac{T}{2},$$ which is not a function of $\theta$, $\theta_0$ or $X$.

Indeed, the two quantities are still not equal after taking expectations: $$\mathbb{E}[l(\hat\theta;X)]=-\frac{T}{2}\log{2\pi}-\frac{T}{2}(\theta_0^2+1)+\frac{1}{2T}(T^2\theta_0^2+T)=\mathbb{E}[l(\theta;X)]+\frac{1}{2}.$$

It's still slightly unsatisfying though because the original model is so clearly under-parameterized (I'm not sure if this can be formalized).

Whether a fully-parameterized example exists is another question, but it's of less immediate relevance to my current enquiry.