Resources for matrix calculus for optimization

Question

I'm a grad student trying to absorb from the book Pattern Recognition and Machine Learning. However, I found that I really need a good grasp of matrix calculus before I can deduct the formulas myself (since I think in this way learning could be more effective). For example, when deducting Gaussian Mixture Model using EM algorithm, I can't really do the M-step on my own since I don't know how to do calculus on vectors/matrix.

What I wish to solve is to be able to solve derivatives like $x^TAx$ with respect to $x$.

I did read wikipedia and know the basics of the idea of taking derivatives with respect to vector, but I hope to get a sense and make myself more confirmable with these operations.

Questions:

1) Does people memorize the derivatives of the basic formulas (like $x^TAx$, or derivaitive of covariance) and then expand?
2) What good resources especially books with practices problems could be recommended?

Thanks a lot

Answer 1

I think a better book is Matrix Calculus by the same Jan Magnus (with H. Neudecker). It goes a little deeper into theory than the Matrix Algebra which is essentially a set of exercises (very good ones, but still... little room for the proofs and discussion of where the math stuff applies.) I heard that the first edition was printed on very low quality paper; don't know about the second edition really as what I have is a Russian edition that I myself helped translating.

Answer 2

A good paper to read is Dwyer (1967) "Some Applications of Matrix Derivatives in Multivariate Analysis". It covers matrix calculus from the perspective of statistical applications. In particular it contains two tables of identities which makes it a useful reference to have on hand.

The paper is available behind a pay wall here or there is currently a freely accessible version on bitbucket.org here although I am unsure of the stability of that link.