# Learning to write your own estimation algorithms - where to start?

I am trained as an epidemiologist. Early in my career, statistical models were a black box, and when in doubt I was advised to go see a statistician. This never sat well with me, so I became interested in statistical modelling and since have put in time learning statistics and probability theory. I am now able to learn, understand, and apply statistical methods that are new to me, but for the first time I have found a model that will likely help with a problem I am dealing with, but as far as I can tell there is no existing software to estimate the parameters of this model. This entire area of statistical modelling is actually a complete mystery to me, and I'd like it not to be.

So...which resources are the most appropriate first step for a novice who wants to learn how to write their own programs for estimating statistical models?

-
What exactly are you modelling, and which probability distribution are you using? –  RobertF Apr 9 at 3:50

As a general thing, this is a pretty broad question (and so may not speak to your immediate, unstated problem), but I'll do my best.

The common approaches to estimation of parameters are maximum likelihood (ML) and method-of-moments (MoM), but there are variations of these, and some that aren't really all that close to either.

Most estimation problems are, or can be cast as optimization or root-finding problems in some guise.

ML seen directly is an optimization problem, while MoM is a root-finding problem (find the parameter values that solve a system of - possibly nonlinear - equations). [Note, however, that if you can take derivatives, ML can also be cast as root-finding.]

Solution techniques for the following are worth knowing about:

• linear least squares problems (e.g. Cholesky, singular value, and QR decompositions)

• weighted linear least squares problems (ditto)

• nonlinear least squares (and weighted nls) (e.g. iterative reweighted least squares)

• ML for generalized linear models (e.g. Fisher scoring)

• ML more generally (e.g. E-M algorithm; there's a host of other optimization/root-finding techniques used here)

Some useful optimization/root-finding/estimation techniques besides those already mentioned, very much incomplete:

• Newton-Raphson

• quasi-Newton methods (BFGS and DFP, for example, also secant method for 1-d)

• steepest descent

• downhill simplex method (aka Nelder-Mead, amoeba)

• golden section

• binary search

$\quad\vdots$

Oh, heck, I haven't even touched on things like Levenberg-Marquardt yet. I may have to come back and add a bunch more at some point.

-
maybe I should also mention KF, BHHH –  Glen_b Apr 9 at 5:34
Thanks. I know I left it broad, perhaps too broad for CV, but I want to be able to solve other problems down the road, not just the one staring me in the face. Is there a particularly valuable text that covers the "most important" methods? –  D L Dahly Apr 9 at 8:59
DL - I sure don't know of any single text that covers all of that (plus the stuff I haven't even mentioned yet) in sufficient depth; that's more like half a dozen books. If there's particular things you want to now about, I may be able to say more. –  Glen_b Apr 9 at 9:11
The key problem is that what is "most important" depends on what you want to do, which you did not specify. –  Maarten Buis Apr 9 at 10:18
Fair enough. I am looking into "latent budget" or "end-member" analysis, which is a sort of reformulation of the latent class model that might be useful for clustering compositional data. Apparently they are "usually estimated by maximum likelihood under the assumption that the frequencies are generated by a product multinomial distribution" (from Ch 4 of Applied Latent Class Analysis - Hagenaars - which can still be found online ewd.zozlak.org/ZlozoneModeleSkalowaniaLiiniowego/…) –  D L Dahly Apr 9 at 10:24