# What is the definition of machine learning (vs classical statistics), and can methods such as MCMC and bootstrapping be considered ML? [duplicate]

I'm in the process of writing about the differences between machine learning and classical statistics. I've been looking for some authoritative sources that would give a good, clear, plain-English definition, but I'm struggling to find anything I like.

Personally, my own, simplest definition of classical statistics would be something like:

"Using formal mathematical proofs and assumptions to model processes underlying data and use these for inference and/or prediction"

From my experience, the only fundamental difference between ML and statistics is that ML skirts the more complicated mathematics by relying on iteration - having a computer do something repeatedly, over and over again.

One often cited difference is that classical statistics focuses on inference whereas ML focuses on prediction, but that's not an essential difference and from what I can tell, it's less and less true over time. Especially recently, many of the big guys in the field (Friedman, Hastie, Tibshirani) have been promoting statistical learning which synthesizes both classical stats & ML and uses methods from either field for both inference and prediction.

Where I think it get especially tangled is in resampling methods. For example, bootstrapping and permutation tests are used in both classical stats and machine learning. By my own definition, I'd call bootstrapping machine learning, since we can use it to avoid having to do complicated mathematics by iterating a simple algorithm (repeatedly drawing random resamples of the original data). Similarly, MCMC and HMC methods used in Bayesian statistics rely on iteration to avoid having to compute a multiple integral, so I'd call them in essence "machine learning" too.

So do you know any good sources for what is the fundamental difference between classical stats vs machine learning? Especially if there is a discussion on bootstrapping/permutation/MCMC as machine learning?

• Does this answer your question? The Two Cultures: statistics vs. machine learning? – Bayequentist Jan 9 '20 at 3:58
• I think the second highest answer in that link nails it. You could make the case that any algorithm that “learns” from data is machine learning - linear regression, even just calculating a mean and standard deviation. The difference between the two fields is focus. Statistics of very much about inference and understanding, whereas machine learning is very much about prediction. Although, of course, there is overlap. But these differences are pretty much the root of all the other differences described in other answers in that link. – Mooks Jan 9 '20 at 8:47

In my view, MCMC/bootstrapping/permutation methods all fall under the category of computational techniques. They aren't tied down to a specific approach or way of thinking about a problem but rather an algorithmic approach to a class of problems. Techniques that involve resampling and iteration don't arise from a machine learning framework, they come out of mathematical theory; the main factor in their recent popularity in solving more classical statistical problems is simply computing power, not something borrowed from machine learning. There is very little in machine learning that cannot be motivated in some way from classical statistics and the related mathematics.

I think it will always be easy to identify certain approaches that are "pure" machine learning, especially deep learning approaches, and more generally the "black box" machine learning approaches that are solely concerned with prediction. There will always be classical statistical approaches that don't relate to machine learning in any way. However, trying to draw any distinct boundary between them in the gray area is as fraught as trying to discriminate physics and chemistry where they intersect.

• So you don't buy the argument that machine learning is using iteration to avoid/reduce the complexity of mathematics? Take the deep learning, for example. If we have a specific deep learning problem X - e.g. recognizing images of dogs - I don't see a reason why an analytical shouldn't exist. The issue is that the analytical solution solution to X is so complex it would not be possible to for a human to figure it out. However, you can use iteration to train a deep net & essentially approximate the analytical solution with hierarchical levels of "representation", i.e. tensor operations. – Adam B. Jan 9 '20 at 20:29
• @AdamB. I'm not saying that machine learning doesn't include "using iteration to avoid/reduce the complexity of mathematics" - I'm saying that "using iteration" is not sufficient to make something "machine learning" as distinct from other forms of statistics. I think Mooks' suggestion to see the 2nd answer to the liked duplicate is good - I'd agree with what is written there, too. – Bryan Krause Jan 9 '20 at 21:41
• Why not? Would you say that e.g. bootstrapping & permutation tests are more similar to classical statistics methods rather than to machine learning? But they only became widely accessible with improvements in processing power (e.g. see Efron & Tibshirani, 1991). Outside of that (and MCMC/HMC), what other statistical methods rely on iteration? – Adam B. Jan 9 '20 at 22:08
• @AdamB. The "why not" is the beginning of my answer: "In my view, MCMC/bootstrapping/permutation methods all fall under the category of computational techniques. They aren't tied down to a specific approach or way of thinking about a problem but rather an algorithmic approach to a class of problems". Machine learning definitely does not merely refer to "using a computer". – Bryan Krause Jan 9 '20 at 22:18
• Okay, what about KNN classifiers and hierarchical clustering? Are those also not just algorithmic approaches to a class of problems, and therefore computational techniques & not machine learning? – Adam B. Jan 9 '20 at 23:40

Personally, I find it very hard to draw a line between the two, as there is clearly some overlapping. Machine Learning is a field that is based on classical statistics and USES statistic models heavily. Also, the mathematics behind Machine Learning can get extremely complicated, so I really would not use the mathematical argument as a discriminant.

One important difference, at least to my eyes, is the one of the "modeling vs data-driven". Statistics usually requires the statistician to make assumptions about the structure and/or distribution of the data, trying to guess the relationships between the variables in order to write an appropriate model. A Machine Learning approach, on the other hand, will try to limit assumptions to a minimum and it will let the data "speak for itself".

I will try to give an example with an algorithm that belongs to both statistical and machine learning literature: linear regression.
A statistical approach would be to look at the variables at hand, and based on the knowledge on their meaning, try to understand which ones might interact and which ones might have a non-linear dependency, building a model accordingly.
A fully ML approach would instead be to use a backward elimination process of features starting from a model containing every interaction and every polinomial expansion up to a certain degree, letting the data decide which ones are relevant.

Of course these two approaches meet in the middle most of the time - statisticians also use forward and backward processes to build their models, as well as ML practicioners often work on the feature engineering in order to give them a better meaning.

But this also leads back to the point that you made earlier: Statistics is more often about trying to understand the structure behind the data in an understandable manner, and explainablility is a big factor; Machine Learning on the other hand often focuses more on the prediction, and this allows it to avoid make models that would "oversimplify" the relationships to make them understandable, and instead use the data to infer the most efficient structure possible to forecast new values.

Finally - on bootstrapping, MCMCs and so on: as Bryan mentioned before me, these are computational techniques, and they are used in both approaches. Also Cross Validation is a computational technique that is used in statistics, and the fact that it relies on iterations it does not make it ML.

I would not put a label on every single algorithm, since Statistics and Machine Learning are deeply intertwined and use many common tools, such as the computational techniques that you mentions or many many models, so in the end when you're in the gray area between the two, the fact of "doing Statistics" or "doing Machine Learning" often depends on the mentality that you use when approaching the problem.

• Please, when downvoting try to comment with a comeback or an explanation on why you do not agree, otherwise you're not helping :) – Davide ND Jan 9 '20 at 12:37
• You are right, the mathematics behind machine learning are complicated - I don't doubt that. What I meant to say is that the analytical solution to the given problem is even more complicated. Both stats and ML try to approximate some real $f(x)$ that's hypothesized to have generated the data. Statistics relies on substituting simpler, "good enough" functions that can be understood analytically. ML on the other hand approximates the function by repeatedly performing "simpler" operations (e.g. SGD to minimize loss) until it arrives at an well-performing approximation. – Adam B. Jan 9 '20 at 20:52
• Talking about SGD, to me it seems very similar to MCMC. The only difference is the former is used to find the spot in the parameter space that minimizes the loss function, whereas the latter is used to estimate the shape of the entire probability density over the parameter space. So SGD is "rush to the mode" of the probability density, MCMC is a "gentle stroll" to survey the entire landscape. Sure, loss & probability density aren't the same thing in finite samples, but wouldn't they be in infinite? Wouldn't the shape of the loss function be inverse to the shape of probability density function? – Adam B. Jan 9 '20 at 21:12
• The idea that statistics uses analytical methods while ml uses iteration is not accurate and comes from the fact that classical stats is usually applied to smaller dataset. But even for linear regression, if the data matrix is too big to be inverted for an analytical solution then you HAVE to use iteration to find the values, and yet that does not make it any more ML than statistics. – Davide ND Jan 10 '20 at 9:48
• Also, I can't really see the parallel between SGD and MCMC other than the fact that they are both iterative :) MCMC is more similar to any other Monte Carlo technique at it is based on simulation, while SGD is more numeric optimization and there is no sampling involved – Davide ND Jan 10 '20 at 9:56

Just for the sake of argument, I am putting my two cents here. As I find the answers above/below so far are pretty explanatory. David DN rounded your question up nicely, I think. This subject is very new and therefore, take what you get and run with it.

I worked with stats and I worked in research. I also worked on predictive research. Even the big guys on the market, like YouTube, LinkedIn or any other social media using Algorithm, are not near perfect, because machine learning, although statistics is behind the calculations, all predictive matter is based on human behaviour and such bound to human research first. Then math. Then there is the learned and cultural influences. Once learned, there is a next step. In addition geographically, not everyone on the planet is on the same platform, meaning, statistically the learned differs on human behaviour outcome. And yet, machine learning is pretty intertwined from math, computational, psychology, linguistics, cultural. What would stats mean if there is no story behind it.

I will suggest the approach, rather than thinking of 'the difference' as one or the other, think of the difference as how each field complements each other and what more can be done.

Rather than giving a full response I would like add a factor in the distinction between the two. Let's make the example of a neural network used for classification, most of the times when people get the results they wanted they don't know exactly why they are getting those results. While statistics is more rigorous and always comes with a measure of the confidence that doesn't necessarily happen in ML. You wrote that ML skirts the more complicated mathematics by relying on iteration, but it may also rely on the combination of different algorithms in such a way that it would be difficult to estimate the contribution of each of them, that would be more difficult to justify in statistics, while in ML the main focus is on the results.