Textbooks on Matrix Calculus?

Question

See this question on Math SE.

Short story: I read The Elements of Statistical Learning and got frustrated when I was trying to verify some of the results, e.g., given $$\text{RSS}(\beta) = \left(\mathbf{y}-\mathbf{X}\beta\right)^{T}\left(\mathbf{y}-\mathbf{X}\beta\right)\text{,}$$ then $$\begin{align}&\dfrac{\partial\text{RSS}}{\partial \beta} = -2\mathbf{X}^{T}\left(\mathbf{y}-\mathbf{X}\beta\right) \\ &\dfrac{\partial^2\text{RSS}}{\partial \beta\text{ }\partial \beta^{T}} = 2\mathbf{X}^{T}\mathbf{X}\text{.} \end{align}$$ I am looking for a matrix calculus book which is written like your traditional calculus book (i.e., proofs of theorems, examples, exercises on computation, etc.). I have already seen this question and feel that the text by Magnus and Neudecker focuses too much on the theory, and the text I have by Gentle focuses too little on the theory and too much on the computation side.

Is there a happy medium out there which is accessible to someone with a background in undergraduate analysis?

Answer 1

For most matrix questions I always first refer to "The Matrix Cookbook" (see here).

It is regularly updated due to feedback from various sources. There are proofs contained within, however it is mostly intended as a handbook.

Answer 2

If you found too much theory in the book of Magnus and Neudecker, I recommend this one, also authored by Magnus:

Abadir, K.M. and Magnus, J.R. Matrix Algebra Cambridge University Press, 2005

that has more emphasis on the applications of matrix calculus.

Answer 3

I would highly recommend this 26 pages paper from Stanford University:

"Linear Algebra Review and Reference" by Zico Kolter

It really focus on typical Sum calculations with a lot of i and j everywhere and tells you the corresponding matrix calculation (i.e. using their "vectorized" implementation).

It helps you recognize right away what type of matrix formula you should write to do your calculations.

Answer 4

A user self-deleted the following helpful answer, which I here reproduce in full so that its information is not lost:

You don't really need a lot of results on vector and matrix derivatives for ML, and Tom Minka's paper covers most of it, but the definitive treatment is Magnus & Neudecker's Matrix Differential Calculus with Applications in Statistics and Econometrics.

Indeed, Magnus & Neudecker has excellent reviews on Amazon and Tom Minka's paper (Old and New Matrix Algebra Useful for Statistics, 2000) contains many useful formulas, although he warns "this is advanced material."