References on numerical optimization for statisticians I'm looking for a solid reference (or references) on numerical optimization techniques aimed at statisticians, that is, it would apply these methods to some standard inferential problems (eg MAP/MLE in common models). Things like gradient descent (straight and stochastic), EM and its spinoffs/generalizations, simulated annealing, etc.
I'm hoping it would have some practical notes on implementation (so often lacking in papers). It doesn't have to be completely explicit but should at least provide a solid bibliography. 
Some cursory searching turned up a couple of texts: Numerical Analysis for Statisticians by Ken Lange and Numerical Methods of Statistics by John Monahan. Reviews of each seem mixed (and sparse). Of the two a perusal of the table of contents suggests the 2nd edition of Lange's book is closest to what I'm after. 
 A: James Gentle's Computational Statistics (2009).
James Gentle's Matrix algebra: theory, computations, and applications in statistics‎ (2007), more so towards the end of the book, the beginning is great too but it's not exactly what you're looking for.
Christopher M. Bishop's Pattern Recognition (2006).
Hastie et al.'s The elements of statistical learning: data mining, inference, and prediction‎ (2009).
Are you looking for something as low-level as a text that will answer a question such as: "Why is it more efficient to store matrices and higher dimensional arrays as a 1-D array, and how can I index them in the usual M(0, 1, 3, ...) way?" or something like "What are some common techniques used to optimize standard algorithms such as gradient descent, EM, etc.?"?
Most texts on machine learning will provide in-depth discussions of the topic(s) you're looking for.
A: Nocedal and Wrights book
http://users.eecs.northwestern.edu/~nocedal/book/
is a good reference for optimization in general, and many things in their book are of interest to a statistician. There is also a whole chapter on non-linear least squares.
A: Optimization, by Kenneth Lange (Springer, 2004), reviewed in JASA by Russell Steele. It's a good textbook with Gentle's Matrix algebra for an introductory course on Matrix Calculus and Optimization, like the one by Jan de Leeuw (courses/202B).
A: As a supplement to these, you may find Magnus, J. R., and H. Neudecker (2007). Matrix Calculus with Applications in Statistics and Econometrics, 3rd ed useful albeit heavy. It develops a full treatment of infinitesimal operations with matrices, and then applies them on a number of typical statistical tasks such as optimization, MLE and non-linear least squares. If in the end of the day you will end up figuring out backward stability of your matrix algorithms, good grasp of matrix calculus will be indispensable. I personally used the tools of matrix calculus in deriving asymptotic results in spatial statistics and multivariate parametric models.
