Reference request: Classical statistics for working data scientists

Question

I'm a working data scientist with solid experience in regression, other machine learning type algorithms, and programming (both for data analysis and general software development). Most of my working life has been focused on building models for predictive accuracy (working under various business constraints), and building data pipelines to support my own (and other's) work.

I have no formal training in statistics, my university education focused on pure mathematics. As such have missed out on learning many of the classical topics, especially the various popular hypothesis tests and inferential techniques.

Are there any references for these topics that would be appropriate for someone with my background and level of experience? I can handle (and appreciate) mathematical rigour, and also enjoy algorithmic perspectives. I tend to like references that offer the reader guided exercises, with both (or either) a mathematical and (or) programming focus.

Answer 1

Larry Wasserman's All of Statistics is a nice book for getting a whirlwind tour of mathematical statistics. It was the first book on mathematical statistics I used myself. It includes the classics like hypothesis testing and maximum likelihood estimation, but it also has plenty of coverage of more recently developed but equally important topics like bootstrapping. Wasserman always has one foot in statistics and the other foot in machine learning, which I think all contemporary data analysts should do; if you're only familiar with one field of the two, you're going to be missing a lot. Also, the book has a lot of good exercises.

If you have a background in real analysis and you want the raw, uncut stuff, by which I mean a measure-theoretic treatment of probability and statistics, try Mark J. Schervish's Theory of Statistics. Schervish is half of DeGroot and Schervish, whose less technical book Probability and Statistics is maybe the most popular book on mathematical statistics today. Theory of Statistics is a helpfully talky book for a topic usually reserved for graduate students who are supposed to do all the work themselves. To be quite honest, I found this book very hard (although not as hard as Jun Shao's Mathematical Statistics) and eventually came to feel the immense effort required to master it wasn't a good use of my time as an applied data analyst. But I still learned a lot and came away with a good understanding of what measure theory is and how it can be used to clean up hairy theoretical difficulties that arise in the more naive traditional approach to probability theory. I also came to better appreciate the similarities and differences of exchangeability and independence.

Answer 2

Aside Kodiologist's very good suggestions (+1) I would also recommend looking at the subject of observational studies. I think it is very unappreciated field between data-scientists despite the fact that in many cases the data analysed are of observational nature. I think this is because the bulk of bibliography (especially in Biostatistics) assume at least some quasi-experimental design is already in place. Paul Rosenbaum's books Observational Studies and Design of Observational Studies are some of the most commonly used references.