Generalization of Newton-Raphson method for maximization of log-likelihood?

Question

Suppose $l(b; y)$ is the log-likelihood function we want to maximize, i.e. we want to solve $U = l' = 0$ for the ML estimation of $b$. Using Newton-Raphson method, we have

\begin{align*} b^{(m)} = b^{(m-1)} - \frac{U}{U'}, \end{align*}

This can be again approximated by: \begin{align*} b^{(m)} = b^{(m-1)} - \frac{U}{E(U')}, \end{align*}

If the random variable $y$ is from a distribution belonging to the exponential family, then we have $M = Var(U) = -E(U')$, thus:

\begin{align*} b^{(m)} = b^{(m-1)} + \frac{U}{M}. \end{align*}

If $\mathbf{b}$ is a vector of several parameters, and $\mathbf{U}$ is vector of derivatives where $U_j = \frac{dl}{db_j}$, then it's said the above formula can be generalized to

\begin{align*} \mathbf{b}^{(m)} = \mathbf{b}^{(m-1)} + \mathbf{M}^{-1} \mathbf{U}, \end{align*}

Where $\mathbf{M}$ is the variance-covariance matrix of $\mathbf{U}$

I am wondering how this generalization can be derived, book recommendations are also very appreciated.

Answer 1

This follows from the equality of the expected Hessian of the log-likelihood to the covariance of the score--both are equal to the Fisher information. Under weak regularity conditions (just need to be able to invoke dominated convergence to commute the derivative with the integral), $$ \mathbf M = E \left[ \nabla^2 l(\mathbf b; y) \right] = E \left[ \nabla l(\mathbf b; y) \nabla l(\mathbf b; y)^T \right]. $$ It's important to recognize that the expections in the above quantity are taken with respect to $y$, with $\mathbf b$ fixed at its true value. Then we recognize that $E \left[ \nabla l(\mathbf b; y) \nabla l(\mathbf b; y)^T \right] = \mbox{Cov}\left[\nabla l(\mathbf b; y) \right]$ because the expectation of the score $E\left[ \nabla l(\mathbf b; y) \right]$ is equal to zero.

Statistical Inference, by Casella and Berger is a good reference for this sort of stuff.