I have looked at this answer and am not satisfied with the results. Reference book for linear algebra applied to statistics?

I have briefly looked at two of the books suggested by the answer, the one by Gentle and the other by Harville and both aren't exactly what I am looking for. Gentle is a bit too dense/wordy. Harville is accessible, but he also presents a lot of information, making it tough to sift through and figure out what's important.

I'm looking for a book or guide that can just tell me the most important properties of linear algebra and the consequences/ramifications. I just need to understand linear algebra enough so I can understand other texts and research papers. I am not looking to derive statistical results. I just need to be literate.

For example, people say Calculus is important for statistics and machine learning, but honestly, if you know that differentiation can be used to find minimums, maximums, the slope of the line (and thus allow you to optimize), and integrals can be used to find area under curves (mostly relevant for CDFs) or continuous sums (Bayesian stuff), that's good enough to understand most statistical/ML concepts - at least so far in my experience. I don't need to actually know all the rules and properties of how to find a derivative. I can barely remember the chain rule, but I don't need to now the chain rule to understand statistics. Someone smarter than me was able to find a derivative, I just need to know that calculus was used to get from point A to point B. Ex. Now that we have set up our cost function, we need to minimize it. I understand that differentiation is used to do this, but either a computer (gradient descent) or someone smart was able to actually minimize it.

I am hoping to get a same level of understanding with linear algebra. Harville's book provides many properties, such as many relationships between different matrices, but I am still not seeing the big picture. For example, what does it mean, actually, when two matrices are orthogonal? Sure, I get the definition that $A'A = I$ but what does it mean? Orthogonal relationships tend to show up in PCA, stuff with correlations/covariances, but I am not seeing how the two relate. Same with stuff on rank and what not. I understand the definition that when a matrix is full rank or nonsingular, all of its columns are linearly independent. But what does that mean when I have a dataset or whatever that is nonsingular? How does that factor into stuff like Johansen's test. And obviously eigenvalues and eigenvectors show up a lot, but I have yet to understand what an eigenvalue actually tells me about something, other than that it satisfies the relationship $Ax = lx$.

The only takeaway I have so far from the book is that inverse matrices or generalized inverses can be used to solve a linear system of equations. It would be nice to see an example of this in action, which the book doesn't give. For example, here's a common problem in statistics that involve solving a system of linear equations (maybe OLS estimation). Here would be an example of a coefficient matrix based on some example data. You can see how this is nonsingular, or not, which means.... The computer calculates the inverse of that and this is how the normal equations are solved.

Any suggestions? Thanks!

  • 1
    $\begingroup$ Yes, I suspect you need a LA text good in applications of the latter, like Applied Linear Algebra available for free written by an Electric Engineer web.stanford.edu/~boyd/vmls/vmls.pdf out of Stanford University . $\endgroup$
    – AJKOER
    Jul 6 '20 at 21:57
  • 2
    $\begingroup$ Gilbert Strang's Introduction to Linear Algebra is always very clear, and it looks like it's posted for free (chapter-wise, at least) here $\endgroup$ Jul 6 '20 at 22:35
  • $\begingroup$ Thanks for the suggestions. I'll take a look. I spent the entire weekend trying to just get an overview on linear algebra and came away with little so am pretty frustrated as I was hoping to get a solid enough grasp to comprehend what I am reading by Monday so I can move on to something else I was trying to learn. I can spend 5 minutes reading about a lemma or corollary only to realize that the lemma is common sense, like $k(A^T)$ is equal to $(kA)^T$. $\endgroup$
    – confused
    Jul 6 '20 at 23:18
  • $\begingroup$ Thanks again, I just looked through the two books and they seem to be exactly what I'm looking for. $\endgroup$
    – confused
    Jul 6 '20 at 23:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.