I'm in the process of writing about the differences between machine learning and classical statistics. I've been looking for some authoritative sources that would give a good, clear, plain-English definition, but I'm struggling to find anything I like.
Personally, my own, simplest definition of classical statistics would be something like:
"Using formal mathematical proofs and assumptions to model processes underlying data and use these for inference and/or prediction"
From my experience, the only fundamental difference between ML and statistics is that ML skirts the more complicated mathematics by relying on iteration - having a computer do something repeatedly, over and over again.
One often cited difference is that classical statistics focuses on inference whereas ML focuses on prediction, but that's not an essential difference and from what I can tell, it's less and less true over time. Especially recently, many of the big guys in the field (Friedman, Hastie, Tibshirani) have been promoting statistical learning which synthesizes both classical stats & ML and uses methods from either field for both inference and prediction.
Where I think it get especially tangled is in resampling methods. For example, bootstrapping and permutation tests are used in both classical stats and machine learning. By my own definition, I'd call bootstrapping machine learning, since we can use it to avoid having to do complicated mathematics by iterating a simple algorithm (repeatedly drawing random resamples of the original data). Similarly, MCMC and HMC methods used in Bayesian statistics rely on iteration to avoid having to compute a multiple integral, so I'd call them in essence "machine learning" too.
So do you know any good sources for what is the fundamental difference between classical stats vs machine learning? Especially if there is a discussion on bootstrapping/permutation/MCMC as machine learning?