Given a pair of RVs $z,\theta$ governed by a joint distribution $p(z,\theta)$. Conditional expectation of $z$ given $\theta$ defines a deterministic function (called as regression functions) $f(\theta)$ given as

$$ f(\theta) = E[z|\theta]=\int zp(z|\theta)dz $$

We have to find the root $\theta^*$ at which $f(\theta^*)=0$.

Now, from the definition of maximum likelihood solution, MLS $\theta_{ML}$ is a stationary point of the log likelihood function and hence satisfies

$$ \frac{\partial}{\partial \theta}\bigg\{\frac{1}{N}\sum_{n=1}^N \ln p(x_n|\theta)\bigg\} = 0 $$

Exchanging the derivative and the summation and taking the limit $N \to \infty$, we have

$$ \lim_{N\to\infty}\frac{1}{N}\sum_{n=1}^N \frac{\partial}{\partial \theta}\ln p(x_n|\theta) = E\bigg[ \frac{\partial}{\partial \theta}\ln p(x|\theta) \bigg] = 0 $$

Hence, we can conclude that finding the maximum likelihood solution corresponds to finding the root of a regression function. How can we reach at this conclusion?

For details, refer Pattern Recognition - Bishop Section 2.3.5 Page 95


1 Answer 1


The connection given in Bishop (2006) (p. 96) is that he is implicitly taking:

$$z = \frac{\partial}{\partial \theta} \ln p(X|\theta),$$

which then gives:

$$\mathbb{E} \bigg[ \frac{\partial}{\partial \theta} \ln p(X|\theta) \bigg| \theta \bigg] = \mathbb{E}(z|\theta) = f(\theta).$$

Once you have that equivalence, and you also apply the law of large numbers to write the expectation as an infinite limiting mean, it then stands to reason that you get the logical equivalence:

$$f(\theta_*)=0 \quad \quad \quad \iff \quad \quad \quad \lim_{N\to\infty} \frac{\partial}{\partial \theta} \frac{1}{N}\sum_{n=1}^N \ln p(x_n|\theta) = 0.$$

This is how he establishes the asserted equivalence. (Note that there are a few implicit assumptions in his working that he does not spell out; in particular, the working assumes that the MLE falls at a critical point of the likelihood function rather than a boundary point.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.