Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am trying to self-teach probability and statistics for Machine Learning career. However I want to learn very well as doing research in AI is my goal.
Which books should I use to learn probability, frequentist statistics and Bayesian statistics and which one should I go first ?
I also know some basic probability, the better route may be to read some frequentist statistical book and then to Bayesian ?
Feel free to suggest any route and book?
If you were an academic, one must assume you already have a good reference for multivariable calculus, linear algebra, and differential equations – these are not optional. I personally heard from Witten and Tibshirani that their texts have the greatest value in working out the problems at excruciating detail including intensive matrix algebra. So, bone up on these skills if you haven't already.
A mathematical pedagogy is fundamentally different from computer science. Whereas CS advocates a top-down approach, mathematics is about finding generalizations. That's why (on this site and elsewhere) you have many self proclaimed "ML experts" who have fit enough algorithms on Kaggle to burn out a network of NVidia graphics cards, but who can't write down an estimating equation to save their life.
If you were a diligent student, you would hope to cover all this over the course of 4-6 years of dedicated study.
If you were a graduate statistics student, you would write a theory course from, say, Casella Berger (research other posts on this one, there may be better texts), linear modeling, and then advanced theory up to minimax estimation, empirical processes, etc. Texts might include Ferguson's A Course in Large Sample Theory, or Lehman, Casella's Theory of Point Estimation. At that point you can read and understand foundational work. These are necessary to "prove" that many algorithmic solutions are well motivated such as the bootstrap, LARS, etc. Referring to "Bayesian" alone is a forgivable newbie mistake, but to participate meaningfully on this site, you need to be more precise. Peter Hoffs "A First Course in Bayesian Statistics" should cover a broad number of areas. Harrell's "Regression Modeling Strategies" is an applied text with some modern solutions that provide a lot of area for research.
Take a look at this page from Arcones and Gine regarding bootstrapping. A procedure as simple as resampling rows with replacement from a dataset repeatedly requires knowledge in a practically completely new area of statistics, empirical process theory. (see texts from Van Der Vaart and Wellner for a reference on this... not for faint of heart!).
If you want to understand the mettle that these researchers bring to the theoretical forefront, you just need to look up any related article on premier research journals, such as biometrika, JRRS, JASA, etc. It is a good exercise at times to find a journal article you really want to understand that's way beyond your ability and try to replicate the results, looking up cited references as needed. With Sci-Hub this is within almost anyone's reach.