Suppose $l(b; y)$ is the log-likelihood function we want to maximize, i.e. we want to solve $U = l' = 0$ for the ML estimation of $b$. Using Newton-Raphson method, we have

\begin{align*} b^{(m)} = b^{(m-1)} - \frac{U}{U'}, \end{align*}

This can be again approximated by: \begin{align*} b^{(m)} = b^{(m-1)} - \frac{U}{E(U')}, \end{align*}

If the random variable $y$ is from a distribution belonging to the exponential family, then we have $M = Var(U) = -E(U')$, thus:

\begin{align*} b^{(m)} = b^{(m-1)} + \frac{U}{M}. \end{align*}

If $\mathbf{b}$ is a vector of several parameters, and $\mathbf{U}$ is vector of derivatives where $U_j = \frac{dl}{db_j}$, then it's said the above formula can be generalized to

\begin{align*} \mathbf{b}^{(m)} = \mathbf{b}^{(m-1)} + \mathbf{M}^{-1} \mathbf{U}, \end{align*}

Where $\mathbf{M}$ is the variance-covariance matrix of $\mathbf{U}$

I am wondering how this generalization can be derived, book recommendations are also very appreciated.


1 Answer 1


This follows from the equality of the expected Hessian of the log-likelihood to the covariance of the score--both are equal to the Fisher information. Under weak regularity conditions (just need to be able to invoke dominated convergence to commute the derivative with the integral), $$ \mathbf M = E \left[ \nabla^2 l(\mathbf b; y) \right] = E \left[ \nabla l(\mathbf b; y) \nabla l(\mathbf b; y)^T \right]. $$ It's important to recognize that the expections in the above quantity are taken with respect to $y$, with $\mathbf b$ fixed at its true value. Then we recognize that $E \left[ \nabla l(\mathbf b; y) \nabla l(\mathbf b; y)^T \right] = \mbox{Cov}\left[\nabla l(\mathbf b; y) \right]$ because the expectation of the score $E\left[ \nabla l(\mathbf b; y) \right]$ is equal to zero.

Statistical Inference, by Casella and Berger is a good reference for this sort of stuff.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.