Distribution for which the log-likelihood evaluated at the ML estimate is not equal to the expected log-likelihood evaluated at the ML estimate

Question

Suppose that for $t=1,\dots,T$, $x_t$ is an i.i.d. draw from a continuous distribution with p.d.f. $f(x_t;\theta_0)$.

Let $l(\theta;X):=\sum_{t=1}^T{\log{f(x_t;\theta)}}$ be the corresponding log-likelihood.

Let $\hat\theta:=\arg\max_\theta{l(\theta;X)}$ be the corresponding ML estimate.

Finally, define:

$$e:=l(\hat\theta;X)-\left.\left[\mathbb{E}l(\theta;X)\right]\right|_{\theta_0=\hat\theta, \theta=\hat\theta}$$

Now, for fairly standard distributions, $e=0$. E.g. for the normal distribution: $$l(\hat\theta;X)=-\frac{T}{2}\log{2\pi}-T\log\hat\sigma-\frac{T}{2}=\left.\left[\mathbb{E}l(\theta;X)\right]\right|_{\theta_0=\hat\theta, \theta=\hat\theta}.$$

Is this really a general property though? I suppose it is equivalent to the assertion that the log-likelihood evaluated at the ML estimate is only a function of the data via $\hat\theta$. This sounds a bit like $\hat\theta$ being a sufficient statistic, so perhaps it is only a property of distributions from the exponential family. (Side question: is this a theorem?)

My main questions then are the following:

Can you find a distribution $f$ for which $e\ne 0$?

Does one such $f$ exist for which the ML estimator admits a closed form solution?

Answer 1

To re-examine the Uniform distribution example given by @jbowman, consider any i.i.d. sample $\{\mathbf x_i;i=1,...,n\}$ from a continuous uniform distribution $U(0,\theta)$. The log-likelihood function is

$$\ell (\theta;\mathbf x_i) = -n\ln \theta + \ln\left (I_{\{ 0\leq x_i \leq \theta;\forall i\}}\right)$$

To consider non-trivially the expected value of the log-likelihood, we have to treat the $x_i$'s as random variables, not as realized values from a specific sample. On the other hand, assuming that $\theta_0 = \hat \theta$, means that $\hat \theta$, although it will be calculated using the $x's$, it should be treated as a constant in the log-likelihood. Then

$$\mathbb{E}\left[\ell(\theta;\mathbf x_i)\middle|\theta_0=\hat\theta, \theta=\hat \theta \right] = -n\mathbb{E}_{\hat \theta}\left[\ln \hat \theta+\ln\left (I_{\{ 0\leq x_i \leq \hat \theta;\forall i\}}\right)\right]$$

But since we calculate this expected value under the assumption that $\hat \theta$ is the true value, it follows that, even though the $x_i$'s are viewed as random variables, we have $I_{\{ 0\leq x_i \leq \hat \theta;\forall i\}}=1$ always. So we end up with

$$\mathbb{E}\left[\ell(\theta;\mathbf x_i)\middle|\theta_0=\hat\theta, \theta=\hat \theta\right]= -n\mathbb{E}_{\hat \theta}[\ln \hat \theta]=-n\ln \hat \theta=\ell (\hat \theta;\mathbf x_i)$$

So we obtain here too the same equality result.

Answer 2

The short answer to both your questions is "yes".

Consider the uniform distribution over $(0, \theta), \theta = 1$. $\mathbb{E}[l(\theta; X)|\theta = 1] = 0$, for obvious reasons. However, since $\hat{\theta} = \max(x_1, \dots, x_n) < 1$, $l(\hat{\theta};X) = \log(1/\hat{\theta}) > 0$.

Your conjecture that it is only a property of distributions from the exponential family is an interesting one, but I am not aware of any results along those lines (although there are probably > 10,000 people who would be more likely than I to be aware of such results.)

Answer 3

I found a simple example that works at last, albeit a slightly unsatisfying one.

Suppose $x_t\sim\mathrm{NIID}(\theta_0,1)$. Then: $$l(\theta;X)=-\frac{T}{2}\log{2\pi}-\frac{1}{2}\sum_{t=1}^T{\left(x_t-\theta\right)^2},$$ so: $$\begin{align}l(\hat\theta;X) & =-\frac{T}{2}\log{2\pi}-\frac{1}{2}\sum_{t=1}^T{\left(x_t-\frac{1}{T}\sum_{s=1}^T{x_s}\right)^2} \\ & =-\frac{T}{2}\log{2\pi}-\frac{1}{2}\sum_{t=1}^T{x_t^2}+\frac{1}{2T}\sum_{s=1}^T{\sum_{t=1}^T{x_s x_t}}.\end{align}$$ However: $$\mathbb{E}[l(\theta;X)]=-\frac{T}{2}\log{2\pi}-\frac{T}{2},$$ which is not a function of $\theta$, $\theta_0$ or $X$.

Indeed, the two quantities are still not equal after taking expectations: $$\mathbb{E}[l(\hat\theta;X)]=-\frac{T}{2}\log{2\pi}-\frac{T}{2}(\theta_0^2+1)+\frac{1}{2T}(T^2\theta_0^2+T)=\mathbb{E}[l(\theta;X)]+\frac{1}{2}.$$

It's still slightly unsatisfying though because the original model is so clearly under-parameterized (I'm not sure if this can be formalized).

Whether a fully-parameterized example exists is another question, but it's of less immediate relevance to my current enquiry.