3
$\begingroup$

Suppose that for $t=1,\dots,T$, $x_t$ is an i.i.d. draw from a continuous distribution with p.d.f. $f(x_t;\theta_0)$.

Let $l(\theta;X):=\sum_{t=1}^T{\log{f(x_t;\theta)}}$ be the corresponding log-likelihood.

Let $\hat\theta:=\arg\max_\theta{l(\theta;X)}$ be the corresponding ML estimate.

Finally, define:

$$e:=l(\hat\theta;X)-\left.\left[\mathbb{E}l(\theta;X)\right]\right|_{\theta_0=\hat\theta, \theta=\hat\theta}$$

Now, for fairly standard distributions, $e=0$. E.g. for the normal distribution: $$l(\hat\theta;X)=-\frac{T}{2}\log{2\pi}-T\log\hat\sigma-\frac{T}{2}=\left.\left[\mathbb{E}l(\theta;X)\right]\right|_{\theta_0=\hat\theta, \theta=\hat\theta}.$$

Is this really a general property though? I suppose it is equivalent to the assertion that the log-likelihood evaluated at the ML estimate is only a function of the data via $\hat\theta$. This sounds a bit like $\hat\theta$ being a sufficient statistic, so perhaps it is only a property of distributions from the exponential family. (Side question: is this a theorem?)

My main questions then are the following:

Can you find a distribution $f$ for which $e\ne 0$?

Does one such $f$ exist for which the ML estimator admits a closed form solution?

Improve this question
$\endgroup$
5
  • $\begingroup$ I am confused by the "double conditioning" of the expectation. You write $|\theta_0 = \hat \theta$ . This could be translated "if it so happens that the obtained estimate equals the true value exactly". Is this the intended meaning? Also, you then add $|\theta = \hat \theta$. In all? $\endgroup$
    – Alecos Papadopoulos
    May 10, 2017 at 19:40
  • $\begingroup$ You were right to query this, as what I'd written was not as clear as it should have been. I have edited the post now, and this alternative notation is I think clearer. Note that the double conditioning is needed as $\mathbb{E}[l(\theta,X)]$ is a function of both $\theta$ (explicitly) and $\theta_0$ (as the distribution of $X$ is). $\endgroup$
    – cfp
    May 11, 2017 at 21:28
  • $\begingroup$ Indeed it is clearer now. I think I will update my answer accordingly $\endgroup$
    – Alecos Papadopoulos
    May 11, 2017 at 21:37
  • $\begingroup$ @AlecosPapadopoulos I'm glad it's clearer. It would certainly be much appreciated if you could update your answer. $\endgroup$
    – cfp
    May 13, 2017 at 16:38
  • $\begingroup$ I am thinking about your other comment also, below my answer. $\endgroup$
    – Alecos Papadopoulos
    May 13, 2017 at 17:28

3 Answers 3

Reset to default
2
$\begingroup$

To re-examine the Uniform distribution example given by @jbowman, consider any i.i.d. sample $\{\mathbf x_i;i=1,...,n\}$ from a continuous uniform distribution $U(0,\theta)$. The log-likelihood function is

$$\ell (\theta;\mathbf x_i) = -n\ln \theta + \ln\left (I_{\{ 0\leq x_i \leq \theta;\forall i\}}\right)$$

To consider non-trivially the expected value of the log-likelihood, we have to treat the $x_i$'s as random variables, not as realized values from a specific sample. On the other hand, assuming that $\theta_0 = \hat \theta$, means that $\hat \theta$, although it will be calculated using the $x's$, it should be treated as a constant in the log-likelihood. Then

$$\mathbb{E}\left[\ell(\theta;\mathbf x_i)\middle|\theta_0=\hat\theta, \theta=\hat \theta \right] = -n\mathbb{E}_{\hat \theta}\left[\ln \hat \theta+\ln\left (I_{\{ 0\leq x_i \leq \hat \theta;\forall i\}}\right)\right]$$

But since we calculate this expected value under the assumption that $\hat \theta$ is the true value, it follows that, even though the $x_i$'s are viewed as random variables, we have $I_{\{ 0\leq x_i \leq \hat \theta;\forall i\}}=1$ always. So we end up with

$$\mathbb{E}\left[\ell(\theta;\mathbf x_i)\middle|\theta_0=\hat\theta, \theta=\hat \theta\right]= -n\mathbb{E}_{\hat \theta}[\ln \hat \theta]=-n\ln \hat \theta=\ell (\hat \theta;\mathbf x_i)$$

So we obtain here too the same equality result.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Having clarified the conditioning in the question, I'm not sure this works actually. $l(\theta,X)=-n\log\theta+log(I_{\hat\theta \le \theta})$, so $\mathbb{E}l(\theta,X)=-n\log\theta+\mathbb{E} log(I_{\hat\theta \le \theta}) = -n\log\theta + 0$ if $ \theta_0 \le \theta $, $ -\infty$ otherwise. Thus, if $\theta_0=\theta$, the second term drops out, so $\left.\left[\mathbb{E}l(\theta,X)\right]\right|_{\theta_0=\hat\theta,\theta=\hat\theta}=-n\log\hat\theta=l(\hat\theta,X)$. $\endgroup$
    – cfp
    May 11, 2017 at 22:04
  • $\begingroup$ @cfp You are right. $\endgroup$
    – Alecos Papadopoulos
    May 13, 2017 at 18:59
1
$\begingroup$

The short answer to both your questions is "yes".

Consider the uniform distribution over $(0, \theta), \theta = 1$. $\mathbb{E}[l(\theta; X)|\theta = 1] = 0$, for obvious reasons. However, since $\hat{\theta} = \max(x_1, \dots, x_n) < 1$, $l(\hat{\theta};X) = \log(1/\hat{\theta}) > 0$.

Your conjecture that it is only a property of distributions from the exponential family is an interesting one, but I am not aware of any results along those lines (although there are probably > 10,000 people who would be more likely than I to be aware of such results.)

Improve this answer
$\endgroup$
2
  • $\begingroup$ But $\left. \mathbb{E}[l(\theta;X) \mid \theta_0=\hat\theta] \right|_{\theta = \hat\theta}$ is different from $\mathbb{E}[l(\theta;X) \mid \theta=1]$ which is what you calculated. The former is the expected value of the likelihood (evaluated at $\hat\theta$) in the hypothetical case that $\hat\theta$ was the true parameter. $\endgroup$
    – Bridgeburners
    May 10, 2017 at 19:39
  • $\begingroup$ Bridgeburners is correct, but from Alecos Papadopoulos's answer above, it does look like jbowman found a great example. $\endgroup$
    – cfp
    May 11, 2017 at 21:02
1
$\begingroup$

I found a simple example that works at last, albeit a slightly unsatisfying one.

Suppose $x_t\sim\mathrm{NIID}(\theta_0,1)$. Then: $$l(\theta;X)=-\frac{T}{2}\log{2\pi}-\frac{1}{2}\sum_{t=1}^T{\left(x_t-\theta\right)^2},$$ so: $$\begin{align}l(\hat\theta;X) & =-\frac{T}{2}\log{2\pi}-\frac{1}{2}\sum_{t=1}^T{\left(x_t-\frac{1}{T}\sum_{s=1}^T{x_s}\right)^2} \\ & =-\frac{T}{2}\log{2\pi}-\frac{1}{2}\sum_{t=1}^T{x_t^2}+\frac{1}{2T}\sum_{s=1}^T{\sum_{t=1}^T{x_s x_t}}.\end{align}$$ However: $$\mathbb{E}[l(\theta;X)]=-\frac{T}{2}\log{2\pi}-\frac{T}{2},$$ which is not a function of $\theta$, $\theta_0$ or $X$.

Indeed, the two quantities are still not equal after taking expectations: $$\mathbb{E}[l(\hat\theta;X)]=-\frac{T}{2}\log{2\pi}-\frac{T}{2}(\theta_0^2+1)+\frac{1}{2T}(T^2\theta_0^2+T)=\mathbb{E}[l(\theta;X)]+\frac{1}{2}.$$

It's still slightly unsatisfying though because the original model is so clearly under-parameterized (I'm not sure if this can be formalized).

Whether a fully-parameterized example exists is another question, but it's of less immediate relevance to my current enquiry.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.