# Confusion over Fisher-scoring algorithm

Given a probability model $$f(X;\theta)$$ and a set of i.i.d. observations $$x_1,\ldots,x_n$$ which we assume to be drawn from some true parameter $$f(X; \theta_0)$$, we can perform maximum-likelihood estimation using Newton's method: $$\theta_{t+1} = \theta_t + \eta (\nabla^2 \ell(\theta_t))^{-1} \nabla \ell(\theta_t)$$ where $$\eta$$ denotes a step-size and $$\ell(\theta) = \sum_{i=1}^n \log f(x_i;\theta)$$

On the other hand, the Fisher-scoring algorithm replaces the Hessian of the log-likelihood, $$\nabla^2 \ell(\theta)$$ with the Fisher information matrix $$\mathcal{I}(\theta) = - \mathbb{E}_{X \sim f(X;\theta)} \nabla^2 \ell(\theta)$$

So, the Fisher-scoring algorithm is $$\theta_{t+1} = \theta_t + \eta (\mathcal{I}(\theta_t))^{-1} \nabla \ell(\theta_t)$$

My question is this: Given that, in general, the iterates of the Fisher scoring algorithm $$\theta_t$$ are not close to the MLE and the true parameter $$\theta_0$$, why is the Fisher information matrix $$\mathcal{I}(\theta_t) = - \mathbb{E}_{X \sim f(X;\theta_t)} \nabla^2 \ell(\theta_t)$$ a reasonable approximation for the Hessian? Specifically, it seems to me that taking expectation over the distribution $$X \sim f(X;\theta_t)$$ will not produce a good estimate for the Hessian of the log-likelihood which is evaluated at the observations $$x_1,\ldots,x_n \sim f(X; \theta_0)$$ at all, unless $$\theta_t$$ is already close to the MLE and thus to the true parameter value.

Yes, but that matters less than you might think

1. For canonical-link generalised linear models, which are a very popular special case, the algorithm is exactly Newton-Raphson
2. For regression models more generally, the expected information will scale appropriately with sample size and with the scale of predictor variable.
3. The expected information is at least always positive semidefinite, which helps for robustness even if not for speed
4. You usually aren't that far from the optimum
5. The Newton-Raphson algorithm also isn't guaranteed to work well unless you're already close to the maximum

You can often do better than Fisher scoring with a quasi-Newton algorithm or with a trust-region method or something. It's not a wonderful general-purpose optimisation algorithm. But it's pretty good for generalised linear models and the Cox model and even for many other regression models.

TLDR: always use line-search gradient descent or the BFGS algorithm to find the MLE. Fisher scoring is a bad idea.

To discuss this method, we need to compare it to other methods to find the MLE. Let's talk about numerical optimization.

We have dataset $$x_1, ..., x_n$$ on which we wish to perform maximum-likelihood estimation. That is, we want to find the global maximum of the empirical log-likelihood:

$$E(\theta) = \frac{1}{n} \sum_{i=1}^n \nabla \log f(x_i;\theta)$$

1. From a general point of view, this is hard problem. For example, when there are multiple local minima, we are in trouble.
2. Under a concavity assumption (and note that we can always cross our fingers and hope that the starting point is close enough to the maxima so that we can consider only the concave region around the maxima)
3. Second order methods are typically not good for this problem. What you want is the BFGS algorithm or, at least, a gradient descent + line-search combination.
4. In particular, there is a great illustration, I think it's in the wright book appendix B, of the relative performance of Newton (with the addition of a line-search; it doesn't work without it) and standard gradient descent with line-search. The performance benefits are not important, in that example. The nice properties of newton's method only kick-in when you are very close to the true parameter value.

So that's for the practical aspects of optimizing a function. The Fisher algorithm is another method to optimize a function. It is yet another second-order method and it is very similar to Newton's method except:

• you don't get the ultra-fast convergence at the end, due to the mismatch between the theoretical hessian and the true hessian
• you still pay the price of inverting a matrix
• you need to be able to compute the fisher information instead of just taking a Hessian
• you still need to add a line-search or the method won't work
The only cute point of Fisher scoring is that you can perform a single step (without line-search, but you should always add line-search) of it to transform any consistent estimator of $$\theta$$ into an estimator that is asymptotically equivalent to the MLE in the limit of infinite data. I leave that as an exercise to the reader. I don't think that this has practical consequences since the same is true for Newton's method