In the context of asymptotic results involving the maximum likelihood estimator, the use of the subscript $\theta_0$ in the expectations operator means the following,
$$\mathbb{E}_{\theta_0}[l(\theta; X)] = \mathbb{E}_{X \sim f_X(x; \theta_0)}[l(\theta, X)] = \int l(\theta; x)f_X(x; \theta_0) \space dx,$$$$\mathbb{E}_{\theta_0}[l(\theta; X)] = \mathbb{E}_{X \sim f_X(x; \theta_0)}[l(\theta; X)] = \int l(\theta; x)f_X(x; \theta_0) \space dx,$$
where $f_X$ is the probability density of $X$, parametrised by $\theta_0$, and where I've omitted the rangelimits of integration.
As a point of emphasis, within this frequentist context, you are entirely correct in identifying that we do not assume that $\theta_0$ is in any way a random variable. Nor at any stage would it be coherent to consequently speak of a density function on $\theta_0$ with respect to which we are computing expectations.
In contrast with the widely used notation $\mathbb{E}_X[g(X)] = \mathbb{E}_{X \sim f_X(x)}[g(X)]$, where $f_X$ is again a density, the notation $\mathbb{E}_{\theta_0}[\cdot]$ can take some getting used to, as is the case with most issues concerning notation.