How do iterative methods for solving maximizing likelihood problems work?

Does anyone know about the computational iteration processes for maximum likelihood estimation?

If these set of equations cannot be solved practically then how does the computer solve them?

• What algorithm exactly do you have in mind? There is a number of iterative optimization algorithms that can be used with ML. There are whole books on this topic. Please make your question more specific.
– Tim
Jul 20 '16 at 7:24
• You know I wasn't addressing a specific algorithm. But for instance you can imagine the iteration processes for ML involved in a logistic regression. Jul 20 '16 at 7:30
– Tim
Jul 20 '16 at 7:35
• the algorithm used to find the maximum of the likelihood function ("itteration") is something entirely different from the model (logistic regression). So what Tim was looking for was something like Newton-Raphson, or Berndt-Hall-Hall-Hausman or Davidon-Fletcher-Powell or Broyden-Fletcher-Goldfarb-Shanno. Jul 20 '16 at 7:39
• @R.Hes sorry but this is too broad... There are whole books on optimization like this one available freely online: stanford.edu/~boyd/cvxbook One cannot summarize a whole book and a number of algorithms in an short answer.
– Tim
Jul 20 '16 at 8:20

By necessity I will barely scratch the surface here (to do the topic justice would require a much longer answer), but I think that this is such an important issue that at least an outline of some common approaches should be given, with links to some of the things that are mentioned along the way.

Does anyone know about the computational iteration processes for maximum likelihood estimation?

There's not a single method, but many different methods suitable for different circumstances.

If these set of equations cannot be solved practically then how the computer solves them?

By "practically" I assume you mean something like "algebraically in closed form" -- and strictly speaking the likelihood itself is not solved. Solving is normally something you do with equations, to find arguments that satisfy the equation.

Maximizing the likelihood can sometimes be turned into a problem that involves solving an equation, though.

Recall that the likelihood is a function of the parameters.

Maximizing the likelihood -- finding the parameters that yield the largest value of the likelihood function - is therefore an optimization problem.

There are numerous approaches that are used to optimize functions.

In many cases the log of the likelihood is easier to deal with (for a variety of reasons); the parameters that maximize the log-likelihood will also maximize the likelihood. Further, it's common to take the negative (or sometimes twice the negative) of the log-likelihood and minimize it; in part this is because most optimizers are written as function-minimizers rather than maximizers, though $-2\log\mathcal{L}$ also has important statistical uses.

For functions of continuous random variables it is sometimes possible to use differential calculus to obtain a set of equations whose solution will be turning points of the likelihood function (which can include local maxima; if you can show there's a unique global maximum it will either be at a turning point or a boundary point). However, for most moderately complex problems (and even many fairly simple ones) it's often better to try to maximize the likelihood more directly than try to solve such equations.

More generally, iterative computer methods are used to step through a sequence of parameter-values in such a way as to (if successful) approximately locate the maximum. There are a number of approaches of varying sophistication and requirements. For example, one approach is simply "head downhill" on the likelihood surface (gradient descent/steepest descent), but there's also Newton's method (also see here), various quasi-Newton methods (like BFGS). One approach closely related to Newton's method commonly used to maximize likelihood is Fisher scoring.

These sorts of methods generally require at least first derivatives of the likelihood usually obtained algebraically (though in some cases numeric derivatives are used). Some derivative-free methods exist (like Nelder-Mead, for example). Some methods combine several different techniques, taking advantage of faster methods when they work but using safer-but-slower approaches otherwise.

In general, however, multiple local maxima can occur, and then (at least in most cases) identifying the location of the global maximum may be very difficult.

Here's a simple example, with a small sample from a Cauchy distribution: Sometimes maxima occur at boundaries; this may cause difficulties for many of those methods (at least if applied uncritically).

A common example offered to beginning students is to estimate $\theta$ from a sample from a uniform on $(0,\theta)$ -- just blindly trying to set the derivative of the likelihood (or the log-likelihood) to zero will be unfruitful, but it's a trivial case to optimize with a little care. Sometimes an iterated function ($\theta = F(\theta)$) can be set up such that the maximum of the likelihood corresponds to a fixed point of the function. An example is seen in this answer.

In other circumstances, you may have discrete parameters. One common example is the hypergeometric distribution (the parameter being the number of success states in the population). However in that particular case the likelihood can be shown to be increasing to the left of a particular value, and decreasing to the right of it, making the solution simple for the hypergeometric.

In some discrete cases finding the actual ML may be NP-hard. Sometimes various approximations may be used to get reasonable estimates in reasonable time.

• Vow, it was harder than what I thought. Well, thank you Glen. Jul 20 '16 at 10:52
• @R.Hes If there's something you'd like more information on up there, or clarification needed, let me know and I'll see what I can do. Jul 20 '16 at 10:56
• Thank you man. You proved your genius and kindness. The only thing that I would like to ask is that how can I master statistics. You know I am a medical student, but more interested in research field. I have gone quite descent path through statistics in a way that I have amazed our university statisticians in some way. But the more I go through the concepts and the equation the more I feel that I need to study. So frankly saying, I would really like to be in your shoes. Is there any advice that you could give me. Like I'm interested to know the curriculum for study statistics in university. Jul 20 '16 at 11:04
• Discussion continued here. Jul 25 '16 at 8:36