Learning to write your own estimation algorithms - where to start?

I am trained as an epidemiologist. Early in my career, statistical models were a black box, and when in doubt I was advised to go see a statistician. This never sat well with me, so I became interested in statistical modelling and since have put in time learning statistics and probability theory. I am now able to learn, understand, and apply statistical methods that are new to me, but for the first time I have found a model that will likely help with a problem I am dealing with, but as far as I can tell there is no existing software to estimate the parameters of this model. This entire area of statistical modelling is actually a complete mystery to me, and I'd like it not to be.

So...which resources are the most appropriate first step for a novice who wants to learn how to write their own programs for estimating statistical models?

-
What exactly are you modelling, and which probability distribution are you using? – RobertF Apr 9 '14 at 3:50

As a general thing, this is a pretty broad question (and so may not speak to your immediate, unstated problem), but I'll do my best.

The common approaches to estimation of parameters are maximum likelihood (ML) and method-of-moments (MoM), but there are variations of these, and some that aren't really all that close to either.

Most estimation problems are, or can be cast as optimization or root-finding problems in some guise.

ML seen directly is an optimization problem, while MoM is a root-finding problem (find the parameter values that solve a system of - possibly nonlinear - equations). [Note, however, that if you can take derivatives, ML can also be cast as root-finding.]

Solution techniques for the following are worth knowing about:

• linear least squares problems (e.g. Cholesky, singular value, and QR decompositions)

• weighted linear least squares problems (ditto)

• nonlinear least squares (and weighted nls) (e.g. iterative reweighted least squares)

• ML for generalized linear models (e.g. Fisher scoring)

• ML more generally (e.g. E-M algorithm; there's a host of other optimization/root-finding techniques used here)

Some useful optimization/root-finding/estimation techniques besides those already mentioned, very much incomplete:

• Newton-Raphson

• quasi-Newton methods (BFGS and DFP, for example, also secant method for 1-d)

• steepest descent

• downhill simplex method (aka Nelder-Mead, amoeba)

• golden section

• binary search

$\quad\vdots$