What is the definition of machine learning (vs classical statistics), and can methods such as MCMC and bootstrapping be considered ML? [duplicate]

Question

I'm in the process of writing about the differences between machine learning and classical statistics. I've been looking for some authoritative sources that would give a good, clear, plain-English definition, but I'm struggling to find anything I like.

Personally, my own, simplest definition of classical statistics would be something like:

"Using formal mathematical proofs and assumptions to model processes underlying data and use these for inference and/or prediction"

From my experience, the only fundamental difference between ML and statistics is that ML skirts the more complicated mathematics by relying on iteration - having a computer do something repeatedly, over and over again.

One often cited difference is that classical statistics focuses on inference whereas ML focuses on prediction, but that's not an essential difference and from what I can tell, it's less and less true over time. Especially recently, many of the big guys in the field (Friedman, Hastie, Tibshirani) have been promoting statistical learning which synthesizes both classical stats & ML and uses methods from either field for both inference and prediction.

Where I think it get especially tangled is in resampling methods. For example, bootstrapping and permutation tests are used in both classical stats and machine learning. By my own definition, I'd call bootstrapping machine learning, since we can use it to avoid having to do complicated mathematics by iterating a simple algorithm (repeatedly drawing random resamples of the original data). Similarly, MCMC and HMC methods used in Bayesian statistics rely on iteration to avoid having to compute a multiple integral, so I'd call them in essence "machine learning" too.

So do you know any good sources for what is the fundamental difference between classical stats vs machine learning? Especially if there is a discussion on bootstrapping/permutation/MCMC as machine learning?

Answer 1

In my view, MCMC/bootstrapping/permutation methods all fall under the category of computational techniques. They aren't tied down to a specific approach or way of thinking about a problem but rather an algorithmic approach to a class of problems. Techniques that involve resampling and iteration don't arise from a machine learning framework, they come out of mathematical theory; the main factor in their recent popularity in solving more classical statistical problems is simply computing power, not something borrowed from machine learning. There is very little in machine learning that cannot be motivated in some way from classical statistics and the related mathematics.

I think it will always be easy to identify certain approaches that are "pure" machine learning, especially deep learning approaches, and more generally the "black box" machine learning approaches that are solely concerned with prediction. There will always be classical statistical approaches that don't relate to machine learning in any way. However, trying to draw any distinct boundary between them in the gray area is as fraught as trying to discriminate physics and chemistry where they intersect.

Answer 2

Personally, I find it very hard to draw a line between the two, as there is clearly some overlapping. Machine Learning is a field that is based on classical statistics and USES statistic models heavily. Also, the mathematics behind Machine Learning can get extremely complicated, so I really would not use the mathematical argument as a discriminant.

One important difference, at least to my eyes, is the one of the "modeling vs data-driven". Statistics usually requires the statistician to make assumptions about the structure and/or distribution of the data, trying to guess the relationships between the variables in order to write an appropriate model. A Machine Learning approach, on the other hand, will try to limit assumptions to a minimum and it will let the data "speak for itself".

I will try to give an example with an algorithm that belongs to both statistical and machine learning literature: linear regression.
A statistical approach would be to look at the variables at hand, and based on the knowledge on their meaning, try to understand which ones might interact and which ones might have a non-linear dependency, building a model accordingly.
A fully ML approach would instead be to use a backward elimination process of features starting from a model containing every interaction and every polinomial expansion up to a certain degree, letting the data decide which ones are relevant.

Of course these two approaches meet in the middle most of the time - statisticians also use forward and backward processes to build their models, as well as ML practicioners often work on the feature engineering in order to give them a better meaning.

But this also leads back to the point that you made earlier: Statistics is more often about trying to understand the structure behind the data in an understandable manner, and explainablility is a big factor; Machine Learning on the other hand often focuses more on the prediction, and this allows it to avoid make models that would "oversimplify" the relationships to make them understandable, and instead use the data to infer the most efficient structure possible to forecast new values.

Finally - on bootstrapping, MCMCs and so on: as Bryan mentioned before me, these are computational techniques, and they are used in both approaches. Also Cross Validation is a computational technique that is used in statistics, and the fact that it relies on iterations it does not make it ML.

I would not put a label on every single algorithm, since Statistics and Machine Learning are deeply intertwined and use many common tools, such as the computational techniques that you mentions or many many models, so in the end when you're in the gray area between the two, the fact of "doing Statistics" or "doing Machine Learning" often depends on the mentality that you use when approaching the problem.

Answer 3

Just for the sake of argument, I am putting my two cents here. As I find the answers above/below so far are pretty explanatory. David DN rounded your question up nicely, I think. This subject is very new and therefore, take what you get and run with it.

I worked with stats and I worked in research. I also worked on predictive research. Even the big guys on the market, like YouTube, LinkedIn or any other social media using Algorithm, are not near perfect, because machine learning, although statistics is behind the calculations, all predictive matter is based on human behaviour and such bound to human research first. Then math. Then there is the learned and cultural influences. Once learned, there is a next step. In addition geographically, not everyone on the planet is on the same platform, meaning, statistically the learned differs on human behaviour outcome. And yet, machine learning is pretty intertwined from math, computational, psychology, linguistics, cultural. What would stats mean if there is no story behind it.

I will suggest the approach, rather than thinking of 'the difference' as one or the other, think of the difference as how each field complements each other and what more can be done.

Answer 4

Rather than giving a full response I would like add a factor in the distinction between the two. Let's make the example of a neural network used for classification, most of the times when people get the results they wanted they don't know exactly why they are getting those results. While statistics is more rigorous and always comes with a measure of the confidence that doesn't necessarily happen in ML. You wrote that ML skirts the more complicated mathematics by relying on iteration, but it may also rely on the combination of different algorithms in such a way that it would be difficult to estimate the contribution of each of them, that would be more difficult to justify in statistics, while in ML the main focus is on the results.