Importance of complicated linear algebra for data scientist [closed]

Question

I've already finished Andrew Ng's machine learning course and now working with textbook 'The Elements of Statistical Learning'. I'm successfully implementing equations and concepts described there using MATLAB, but I don't get all the things going on there with linear algebra, like why are degrees of freedom for ridge regression are defined that way or why can we write least squares fitted vector using singular value decomposition another way. I get common sense of that equations and understand where and for what reason they should be used, but I can't perform the same transformations of them, like where they come from.

Is it a big problem for beginning data scientist? I mean, does it appear in practice that junior data scientist should create some formula himself/herself? I know basics of linear algebra, but equations and transformations given there are far higher level than basic or intermediate.

Answer 1

To add to what @Sycorax commented:

If you're satisfied with merely being a consumer of scientific software, you can skate by without much knowledge at all. But if you're curious and interested in really understanding what's going on when your script runs, knowledge of statistics, linear algebra, calculus and numerical optimization are essential; the more, the better.

So specifically linear algebra, what do you need to be a better user, understand software (and methods) better, and for modeling (but not being interested, at this moment at least, in writing software or proving theorems). The linear model is the backbone of much of statistics, so you need to understand that well. Since a linear model can be seen geometrically as a subspace of a certain linear space, and this is just the basics of linear algebra: spaces, subspaces, linear combinations, linear independence (and dependence). You need this to understand the specification of a linear model, and the various problems that can occur.

Then the basics of matrices, linear equations in matrix form, solutions, matrix inverses. And a lot of linear algebra this days in embedded in matrix factorizations, the spectral decomposition, the svd (see What is the intuition behind SVD?), the QR decomposition (see Understanding QR Decomposition.)

I found it useful to get a better intuition for matrix multiplication, see https://math.stackexchange.com/questions/198257/intuition-for-the-product-of-vector-and-matrices-xtax/198280#198280.

So where to learn this? Assuming you have learnt a little linalg in the past, I would start with some linear models book with an appendix an matrices. For instance Linear Regression Analysis (2nd Edition) by Seber and Lee but there are no lack of competition. And read Venables & Ripley MASS (the book, fourth edition) section 6.2 Model Formulae and Model Matrices.

Answer 2

I think linear algebra is absolutely essential to the data scientist. A card carrying data scientist should have more than just a comprehensive knowledge of what tools are out there, but he or she should also have the ability to compare them, to develop new tools, and, when a job simply can't be solved by any tool, be able to explain why.

Linear algebra (and calculus) is useful for the following essential tasks that a data scientist should be able to do:

• Deriving first and second order derivatives and gradients for multivariate expressions
• Solving complex systems of equations
• Comparing the relative efficiency of two estimators
• Showing that an estimator (of a parameter, of a density function) is minimax, unbiased, or consistent
• Understanding the performance and behavior of nonlinear systems and their dependence on initial conditions
• Showing that the inverse gamma prior is conjugate for a Bayesian linear regression
• Showing the n-consistency of order-statistic estimators like the median

As a master level statistician, I have had to use algebra (maths and calculus included) to prove a few things:

1. Derive a variance estimate for a marginal standardization estimate of relative risk from logistic regression
2. Prove that the "information" of a clinical trial vis-a-vis blinded conditional power is the same as sample size.
3. Derive an EM algorithm to maximize likelihood for truncated data and missing data
4. Prove that a bootstrap combined with multiple imputation gives 95% CIS with accurate coverage

Answer 3

I worked with data scientists who do not know linear algebra. The field, which pretentiosly has "science" in its name, is so vast that there's something to do for everyone willing. It is somewhat similar to programmers not knowing electronics, and most of them have no clue. You can survive and even prosper without linear algebra but you will not be able to do certain things, and you'll be slower at certain tasks.

The main area where you will get hurt is research. Ng's lectures show a bit of internals, he often refers to papers, shows equations etc. If you're going to be implementing new algorithm that are just off the press, it would be very difficult without linear algebra to understand a lot of research papers. Here, it will be like coming with a knife to a gun fight, you'll be the dumbest guy in the room almost all the time. It would be depressing.

However, possibly 90% of work in data science is not of this nature. Most of the time you'll be asked to create reports, massage data, and maybe call libraries. This doesn't require any knowledge of anything. Any programmer could do it, if they wanted to spend a couple of weeks on learning the frameworks.