I am trying to study the asymptotic properties of MLE, but I am having trouble understanding an expression that seems to be consistently used in all lecture notes available online (page 93,page 18,page 5):

$$\frac{1}{n}\sum_{i=1}^n l(\theta;X_i) \rightarrow E_{\theta_0}\left[ l(\theta;X) \right]$$

as $n \rightarrow \infty$. $l$ is a likelihood function, $X_i$ is a sample, $\theta_0$ is the ground truth parameter value. In one of the notes, it says $E_{\theta_0}$ means "expectation with respect to true unknown parameter", but what does this exactly mean? When there is a subscript on $E$, I always thought it should be a random variable. For example, $E_X[f(X)]$ means $f(X)$ averaged over a distribution of $X$. But $\theta_0$ (the true parameter value) is not a distribution, but instead, a particular value.

Also, shouldn't it be $E_X$ instead of $E_{\theta_0}$ in the above expression, since we are summing over all possible values of $X$ according to its probability?

  • $\begingroup$ The subscript $\theta_0$ is said to index a family of distributions which are fundamental to the hypothesis being tested. When we consider a broader family of distributions and hypothesis tests, we can use "bigger" language like $E_{F_0}(T(X))$, where the null hypothesis might be $F = F_{0}$ where $F$ is the CDF of $X$. But in your case, if the distribution is, say, exponential, if you give me $\theta$, I know what the distribution is. And if the distribution is gamma, and you only give me the shape, I just maximize the constrained likelihood to find the scale to test hypotheses. $\endgroup$
    – AdamO
    Jul 19 '21 at 19:28

In the context of asymptotic results involving the maximum likelihood estimator, the use of the subscript $\theta_0$ in the expectations operator means the following,

$$\mathbb{E}_{\theta_0}[l(\theta; X)] = \mathbb{E}_{X \sim f_X(x; \theta_0)}[l(\theta; X)] = \int l(\theta; x)f_X(x; \theta_0) \space dx,$$

where $f_X$ is the probability density of $X$, parametrised by $\theta_0$, and where I've omitted the limits of integration.

As a point of emphasis, within this frequentist context, you are entirely correct in identifying that we do not assume that $\theta_0$ is in any way a random variable. Nor at any stage would it be coherent to consequently speak of a density function on $\theta_0$ with respect to which we are computing expectations.

In contrast with the widely used notation $\mathbb{E}_X[g(X)] = \mathbb{E}_{X \sim f_X(x)}[g(X)]$, where $f_X$ is again a density, the notation $\mathbb{E}_{\theta_0}[\cdot]$ can take some getting used to, as is the case with most issues concerning notation.

  • $\begingroup$ @CWC. In response to the comment which has been deleted, concerning whether what I've written is consistent with AdamO's comment. The connection between hypothesis testing and the asymptotics of maximum likelihood estimators is unclear to me. That is not a denial that a connection may exist, rather, I don't often see those areas being weaved together in expository notes and references on the subject,. $\endgroup$
    – microhaus
    Jul 19 '21 at 22:09
  • $\begingroup$ Thank you for the insight! $\endgroup$
    – CWC
    Jul 19 '21 at 23:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.