What does "Expectation with respect to true unknown parameter" mean?

Question

I am trying to study the asymptotic properties of MLE, but I am having trouble understanding an expression that seems to be consistently used in all lecture notes available online (page 93,page 18,page 5):

$$\frac{1}{n}\sum_{i=1}^n l(\theta;X_i) \rightarrow E_{\theta_0}\left[ l(\theta;X) \right]$$

as $n \rightarrow \infty$. $l$ is a likelihood function, $X_i$ is a sample, $\theta_0$ is the ground truth parameter value. In one of the notes, it says $E_{\theta_0}$ means "expectation with respect to true unknown parameter", but what does this exactly mean? When there is a subscript on $E$, I always thought it should be a random variable. For example, $E_X[f(X)]$ means $f(X)$ averaged over a distribution of $X$. But $\theta_0$ (the true parameter value) is not a distribution, but instead, a particular value.

Also, shouldn't it be $E_X$ instead of $E_{\theta_0}$ in the above expression, since we are summing over all possible values of $X$ according to its probability?

Answer 1

In the context of asymptotic results involving the maximum likelihood estimator, the use of the subscript $\theta_0$ in the expectations operator means the following,

$$\mathbb{E}_{\theta_0}[l(\theta; X)] = \mathbb{E}_{X \sim f_X(x; \theta_0)}[l(\theta; X)] = \int l(\theta; x)f_X(x; \theta_0) \space dx,$$

where $f_X$ is the probability density of $X$, parametrised by $\theta_0$, and where I've omitted the limits of integration.

As a point of emphasis, within this frequentist context, you are entirely correct in identifying that we do not assume that $\theta_0$ is in any way a random variable. Nor at any stage would it be coherent to consequently speak of a density function on $\theta_0$ with respect to which we are computing expectations.

In contrast with the widely used notation $\mathbb{E}_X[g(X)] = \mathbb{E}_{X \sim f_X(x)}[g(X)]$, where $f_X$ is again a density, the notation $\mathbb{E}_{\theta_0}[\cdot]$ can take some getting used to, as is the case with most issues concerning notation.