I have been working in R for a bit and have been faced with things like PCA, SVD, QR decompositions and many such linear algebra results (when inspecting estimating weighted regressions and such) so I wanted to know if anyone has a recommendation on a good comprehensive linear algebra book which is not too theoretical but is mathematically rigorous and covers all of these such topics.
12 Answers
$\begingroup$
$\endgroup$
1
The "big three" that I have used/heard of are:
Gentle, Matrix Algebra: Theory, Computations, and Applications in Statistics. (Amazon link).
Searle, Matrix Algebra Useful for Statistics. (Amazon link).
Harville, Matrix Algebra From a Statistician's Perspective. (Amazon link).
I have used Gentle and Harville and found both to be very helpful and quite manageable.
-
4$\begingroup$ There is also F. A. Graybill, Matrices with Applications in Statistics, 2nd. ed., Duxbury, 2001. It has a wealth of good information and serves as a companion volume to Graybill's linear models text. It is, perhaps, slightly dated in that other texts place more emphasis on the power of the SVD. $\endgroup$– cardinalCommented Jan 20, 2012 at 14:45
$\begingroup$
$\endgroup$
The Matrix Cookbook by K. B. Petersen.
is a free resource will all sorts of useful identities involving various decompositions, forms of inverses for various commonly encountered matrix structures, formulas for differentiating matrix functions and much more. You'll probably find whatever you're looking for in the matrix cookbook. I've never found any mistakes at all there, but since the matrix cookbook is a free resource, it is not professionally edited, so there could potentially be errors there. But, it is regularly being updated, so I wouldn't worry too much about that.
Although this is a general purpose manual, there is certainly a statistics slant to it, as you will see.
$\begingroup$
$\endgroup$
Matrix Computations by Golub and Van Loan is the standard reference for matrix computation for many.
$\begingroup$
$\endgroup$
I've found Advanced Multivariate Statistics with Matrices by Kollo and von Rosen to be very useful when working with multivariate statistics. The first 170 pages are linear algebra. It then goes on to cover multivariate distributions, asymptotics and linear models - all in a rigorous way. It doesn't cover projection methods though.
$\begingroup$
$\endgroup$
1
In addition to the three mentioned by @Mike Wierzbicki (all of which I use), another useful one is "Matrix Tricks for Linear Statistical Models" by Puntanen, Styan and Isotalo (2011).
-
$\begingroup$ The must-have reference for orthogonal projections! $\endgroup$– YvesCommented Apr 14, 2023 at 14:47
$\begingroup$
$\endgroup$
You could try "Numerical Methods of Statistics", by John F. Monahan. It assumes that you know linear algebra, but the author's web site provides programs coded in R.
$\begingroup$
$\endgroup$
Krishnan Namboodiri's Matrix Algebra: An Introduction is a quick, bare-bones way to learn much of the linear algebra you'll need.
You can also try MIT OCW.
$\begingroup$
$\endgroup$
I have Anton's Elementary Linear Algebra, mainly for the chapters on linear equations and matrices and on determinants (I have the 7th edition).
$\begingroup$
$\endgroup$
As a mathematical statistics student Rencher's book named Linear Models In Statistics was very helpful for me, especially in working with mean and variance of quadratic forms. It is available in this link. I hope it could be useful for other students and researchers too.
$\begingroup$
$\endgroup$
It doesn't advertise itself as "for statisticians", but many statisticians have made great use of Gil Strang's Intro to Linear Algebra, which covers all the topics you describe, and has chapters about statistical applications.
$\begingroup$
$\endgroup$
Mathematics for Machine Learning is another nice alternative (freely available)
$\begingroup$
$\endgroup$
I second many of the books recommended, especially Rencher's book Linear Models In Statistics. Another book I would recommend is Hands-On Matrix Algebra Using R: Active And Motivated Learning With Applications (amazon link). It is not overly technical, and provides many examples in R, which I found useful when learning.