Question about the Robbins-Monro algorithm for sequential maximum likelihood estimation

Question

I am confused about the derivation in Section 2.3.5 of Pattern Recognition and Machine Learning (Bishop, 2006). The Robbins-Monro algorithm (Eq. 2.129) to find a root $\theta^\star$ of the regression function $\mathbb E[z \mid \theta]$ is: $$ \theta^{(N)} = \theta^{(N - 1)} - a_{N - 1}z(\theta^{(N-1)}), $$ where $N$ is the iteration step and "$z(\theta^{(N)})$ is an observed value of $z$ when $\theta$ takes the value $\theta^{(N)}$". This procedure is adapted to maximum likelihood estimation by letting $z$ be the derivative of the log-likelihood function, such that a root corresponds to the MLE. The procedure (Eq. 2.135) then takes the form $$ \theta^{(N)} = \theta^{(N - 1)} + a_{N - 1} \frac{\partial}{\partial \theta}\ln p(x_N \mid \theta^{(N-1)}). $$ The one point that I am confused about is that I don't see how this choice of $z$ is "an observed value of $z$ when $\theta$ takes the value $\theta^{(N-1)}$". It is a random variable because it depends on $x_N$, but $x_N$ is sampled from a distribution with the true parameter $\theta$, not "when $\theta = \theta^{(N-1)}$". But wouldn't that be required? Thanks!

Answer 1

I think I found the source of my confusion. In this setting, the distribution over $x$ is unrelated to the model distribution $p(x\mid \theta)$, which does not even have to be of the same type as the true $p(x)$. So, as $x_N$ is not sampled from $p(x \mid \theta)$, it can be sampled from the same distribution (or dataset), no matter what value $\theta$ holds. In oher words, the expectation in Eq. (2.134) is not conditioned on $\theta$.