Finding the maximum likelihood solution corresponds to finding the root of a regression function. How?

Question

Given a pair of RVs $z,\theta$ governed by a joint distribution $p(z,\theta)$. Conditional expectation of $z$ given $\theta$ defines a deterministic function (called as regression functions) $f(\theta)$ given as

$$ f(\theta) = E[z|\theta]=\int zp(z|\theta)dz $$

We have to find the root $\theta^*$ at which $f(\theta^*)=0$.

Now, from the definition of maximum likelihood solution, MLS $\theta_{ML}$ is a stationary point of the log likelihood function and hence satisfies

$$ \frac{\partial}{\partial \theta}\bigg\{\frac{1}{N}\sum_{n=1}^N \ln p(x_n|\theta)\bigg\} = 0 $$

Exchanging the derivative and the summation and taking the limit $N \to \infty$, we have

$$ \lim_{N\to\infty}\frac{1}{N}\sum_{n=1}^N \frac{\partial}{\partial \theta}\ln p(x_n|\theta) = E\bigg[ \frac{\partial}{\partial \theta}\ln p(x|\theta) \bigg] = 0 $$

Hence, we can conclude that finding the maximum likelihood solution corresponds to finding the root of a regression function. How can we reach at this conclusion?

For details, refer Pattern Recognition - Bishop Section 2.3.5 Page 95

Answer 1

The connection given in Bishop (2006) (p. 96) is that he is implicitly taking:

$$z = \frac{\partial}{\partial \theta} \ln p(X|\theta),$$

which then gives:

$$\mathbb{E} \bigg[ \frac{\partial}{\partial \theta} \ln p(X|\theta) \bigg| \theta \bigg] = \mathbb{E}(z|\theta) = f(\theta).$$

Once you have that equivalence, and you also apply the law of large numbers to write the expectation as an infinite limiting mean, it then stands to reason that you get the logical equivalence:

$$f(\theta_*)=0 \quad \quad \quad \iff \quad \quad \quad \lim_{N\to\infty} \frac{\partial}{\partial \theta} \frac{1}{N}\sum_{n=1}^N \ln p(x_n|\theta) = 0.$$

This is how he establishes the asserted equivalence. (Note that there are a few implicit assumptions in his working that he does not spell out; in particular, the working assumes that the MLE falls at a critical point of the likelihood function rather than a boundary point.)