Comparing model evaluations of machine learning and statistics

I'm a machine learning practitioner who spends most of my time doing applications via Python in machine vision with neural networks and remote sensing data, and evaluating model performance mostly using cross-validation based techniques etc.

Statistics has been (of course) part of my learning path and I am familiar with the basic concepts of statistics, but I am not actively doing the work of a "pure" statistician, that is, using e.g. R/SAS-software to fit generalized linear models, calculating the $$t/p$$-values, ANOVA and testing statistical significance of model covariates.

Even though I've been exposed to these fields for many years now, it still is not clear to me how (traditional/pure) statistics and machine learning differ from each other. I know that the general concensus on the difference is that statistics cares more about explaining the data, whereas machine learning (ML) is interested in making predictions, even though the differences are sometimes vague and both fields use the same methods.

Now in ML, usually when we have fitted some model into the data, we care about how well the model has learned the underlying pattern from the data, and we measure this learning by using an independent test data set. We never want to have a perfect fit of the model into the training data, because this would lead into overfitting and poor generalization capability. So the relevant question in ML (to my understanding) is never about "how good the fit of the model is to the data?", but rather "how well the model is able to predict new situations, that is, generalize".

When I listen to my statistician colleagues, I notice them talking mostly about the $$t/p$$-values, the $$R^2$$-values, $$F$$-statistics, differences in the means of two groups in ANOVA etc.

To me, from a ML perspective, my statistician colleagues seem to be concerned exactly about the "goodness-of-fit" of their models/covariates into the data and there is no independent test data set, which they use to validate their models. Well, of course, they don't do this, because this is not the goal, but rather to explain the data with the explicit assumptions we have made on the data (normality etc. the usual ones).

Now, I most probably am wrong due to my lack of knowledge on the subject, but it somehow seems odd to me why"pure" statisticians are so interested in these "goodness-of-fit"-statistics, because don't the $$p$$-values, $$R^2$$ etc. essentially in the end measure exactly this, that is the model fit (given that our distribution assumptions are correct)? For example, we all know that neural networks are universal approximators, meaning we can fit them with 100% accuracy into any (continuous) function we arbitrarily choose by just adding enough neurons into the model. Now, wouldn't this universal approximating neural network tuned into our data like hell have a huge statistical significance in the $$p$$-values or $$R^2$$-metrics, if we look at the model fit from a "pure" statistician's perspective? As a summary, would a statistician (does he?) conclude that: "we have found something truly significant now" in this neural network scenario? A ML scientist would produce an independent test sample, feed it to the network, and conclude that the model is overfitting the training data as hell and no pattern has been found. In other words, ML scientist would conclude "nothing significant has been found".

Maybe I asked my question too vaguely, but here it is summarized: Is it true to some extent that statisticians are usually more concerned about the model's goodness-of-fit and the corresponding metrics of significance, and not that much about model's generalization capability, and vice versa for the ML scientists?

Thank you in advance for any answers, which help me clear up my confusion :)

• Re: the neural net example, the same is true of a linear model when p=n (ie, perfect fit) but it's fair to say most statisticians don't consider that a good or useful model. Usually criticism of model complexity is an important part of appraising the test or summary statistics you described, eg using BIC or adjusted $R^2$ rather than purely goodness of fit measures. Nov 5 '21 at 14:16
• @alanocallaghan Okay, thank you for clearing that up. Yes, it's the same in ML and implemented using regularization, that is penalizing "too-complex" models in the fit-procedure. But still, in the end, an independent test data set is always used to assess generalization. Nov 5 '21 at 14:29
• Frank Harrell has a pretty interesting comment about out-of-sample testing in his Regression Modeling Strategies book, something along the lines of "Why are you spending valuable training data on an out-of-sample test?" (It isn't nearly that dismissive of out-of-sample testing, however.) // Mandatory comment about the "two cultures".
– Dave
Nov 5 '21 at 14:31
• you just happen to deal with causal inference statisticians, they're obviously interested in stuff that you mentioned. there's a ton of statisticians who are into predictive modeling and forecasting, like the guys in marketing. so, your generalization is incorrect. Nov 5 '21 at 14:56
• Related to what Aksakal wrote...if the performance of that model is sufficient to make money for your company (or whatever your goal is), then go use it. That would be in line with the famous George Box quote, along the lines of, "All models are wrong, but some are useful." The model I give there is wrong, but if it can make me a squillionaire, I would consider it useful. // If all you consider in that model is something like square loss, and not violations of assumptions, you will miss that you could improve the model by adding curvature.
– Dave
Nov 5 '21 at 15:24

Is it true to some extent that statisticians are usually more concerned about the model's goodness-of-fit and the corresponding metrics of significance, and not that much about model's generalization capability, and vice versa for the ML scientists?

Scientists and analysts using "pure" statistics have recently gotten into some trouble precisely for focusing excessively on metrics of significance. In fact, in 2017, the American Statistical Association held a Symposium on Statistical Inference which to a large extent discussed the over-reliance that many statisticians have on statistical significance and p-values. It's been a fairly serious problem in the sciences recently.

But your question is difficult to answer, mostly because there are still no agreed upon definitions that mark the difference between statistics/machine learning/data science. A carpenter 2000 years ago used a hammer and nails. Now, a carpenter uses a power drill. When the power drill was invented, did the carpenters who started using it rename themselves? Did they also rename "nail" "multi-object concatenation device (MOCD)" and rename "hammer" "manual MOCD implementer?" Obviously not. That would be absurd. Yet machine learning practitioners and data scientists have more or less done this, and it makes statistics and machine learning seem to be more different than they really are.

High computing power, parallel processing, and new methods became available. Now suddenly a "variable" is a "feature!" You no longer recode your variables, you engage in feature engineering. "Pearson's phi" that was used 100 years ago, you say? No, no, no, it is now the MCC! The list goes on. Differences in jargon often do not reflect any real differences in the underlying mathematics or theory.

Having said that, to the extent that the focus of jobs called "statistician" versus "machine learning practitioner" are starting to diverge, I think you'll find that they diverge in largely the ways you expect. "Statistician" jobs tend to be more scientifically conservative, focusing on understanding relationships and testing assumptions. "Machine learning" jobs tend to be in industry where your goal is to deal quickly and efficiently with a lot of data in order to help make decisions that yield a desired result, which is rarely scientific understanding or evidence for/against hypotheses.

• Excellent, thank you for your help! I agree, I like how Prof. Yaser Abu-Mostafa from Caltech put it (roughly) in one of his seminars that there's a LOT of jargon/renaming going on in the field of statistical learning / machine learning communities and these are also partly due to historical reasons. He wrapped all these many disciplines, ML, data mining, pattern recognition, data analytics, statistical learning etc. under this one phrase: "learning from data, this is the goal in all of them", even though the terminology/assumptions might be different. Nov 5 '21 at 15:51
• @StephanKolassa, I've seen both sides of the coin. My background is social science, but I joined a grad program in statistics largely from disgust with the "magic incantation." To my chagrin, many of my peers and even some professors in statistics don't seem to appreciate the danger. In The American Statistic, see Tong (2018), who voices some concerns that I agree with. IMO, many statisticians who ought to know better use computational power as a crutch rather than a tool, and students learn to treat advanced models like magic pills that excuse them from spending a lot of time thinking.
– AJV
Nov 5 '21 at 16:43
• @StephanKolassa Edited to reflect your comment.
– AJV
Nov 6 '21 at 13:45

The answer is no, too many claims in your post, not only to the main question.

1. The difference between ML and stats is arbitrary, superficial, and not important. There is plenty of statisticians that are doing predictive models where the main goal is, of course prediction. Common machine learning methods have been developed by statisticians and published in statistical journals. The most popular ML book is the elements of statistical learning, after all. Also, there is a huge subfield of ML that is concerned about the interpretation of fitted models and the effects of individual variables. I can tell you that (at least contemporary) papers discussing differences between ML and stats are cynical citation grabs and not really worth the time.

2. Both ML and stats are concerned about learning the underlying pattern from the data. In stats, this might be called estimating effects, but it is the same thing.

3. "[in ML] We never want to have a perfect fit of the model into the training data." Sometimes you do (search double descent)

4. Statisticians do care about the generalization. I would even say more than ML people. All these statistical concepts, like controlling for type I error, confidence intervals, credible intervals etc., are there to tell you how well your findings generalize to the population. Now, if a machine learner tests 2 models in a test set and one is better than the other, this does not tell you much about how this finding will generalize to the population unless you calculate some standard errors, p-values or something on that difference. This is usually not the problem in computer vision with hundreds of thousands of examples, where pretty much any difference will be significant, but it is important in smaller problems.

5. If a model has large goodness of fit because it is overfitted, p/t/F values will not be good, and statisticians will not find significant results in that scenario. In this case, variables will be colinear, standard errors will be huge, and nothing will be significant.

6. in stats, we are often not using test sets because these statistical procedures had been developed to be valid without the need of the test set. If I use F-test to compare which model is better, the results will be generalization to the population because that's what the test is for. It will not automatically select the model with a higher R2.

In summary, statisticians are concerned about the generalization, and the common statistical measures and concepts are there for this purpose. On the other hand, ML people are often not interested in generalization because simple CV or test set performance is often not treated as an estimate of the performance in the population,

• I think this is the key misunderstanding in OP's question : "p/t/F values will not be good" Nov 5 '21 at 15:16
• @rep_ho Thank you for your help! I have a question for point 5: I found a comment from Quora by JQ Veentra that: "really low p-values often indicate overfitting. Why? Because we've fit to the training data too well. You can get really, really small p-values with a high order polynomial regression… we don't do that because it doesn't generalize to new data well (which is the definition of overfitting, by the way).". But isn't small p-values usually a good thing from a statisticians perspective? So is JQ Veentra wrong here or do I and him not understand the p-values correctly? Thank you! Nov 5 '21 at 15:46
• @jjepsuomi that comment is not correct, you cannot get small p-values with a very higher order polynomial regression. Each parameter you fit costs you a degree of freedom, which means that it is harder to find a significant effect. Nov 5 '21 at 15:51
• Thank you for the clarification. This is also the problem, you get so many comments online today that it's difficult to separate the facts from fiction sometimes x) Nov 5 '21 at 15:58
• @jjepsuomi but what does happen, and maybe what JQ Venntra mistaken this for, is that if you do test many variables, than just by chance you will get some small p-values. That's why we have e.g., multiple comparison corrections. But that's not because of overfilling. Nov 5 '21 at 16:04

Is it true to some extent that statisticians are usually more concerned about the model's goodness-of-fit and the corresponding metrics of significance, and not that much about model's generalization capability, and vice versa for the ML scientists?

No. Measuring generalisation capability is a large portion of statistics practice. Cross validation and bootstrapping techniques addresses the question of how well statistical models generalise and selecting a model in Occam's sense: A survey of cross-validation procedures for model selection.

One thing seems to diverge significantly with modern Machine Learning (ML) community, unfortunately, that ML community stop practicing Occam's razor. This is partially because deep learning defy the core paradigms, such as phenomenon of double deep descent. ML community try to establish generalisation via test-train curves on detecting overfitting, i.e., single holdout only. Actually in overfitting, we require two models to compare, not only a holdout on a single model. This is best described by Andrew Gelman, see What is overfitting:

Overfitting is when you have a complicated model that gives worse predictions, on average, than a simpler model.

However, there is now a significant research activity in deep learning as well to introduce Occam's razor indirectly via Neural Architecture Search (NAS). It doesn't aim at directly overfitting rather a model compression, but it is actually form of prevention of overfitting in Gelman's definition.

The question is very long, and covering it in full would require a very lengthy answer. So here I will try to provide my view with very brief bullet points.

1. The question over-emphasises the difference between machine learning and statistics. For a good reflection on those I recommend reading the answer by Michael I. Jordan which was given during his Q&A reddit session: link

2. You use "pure" statisticians in an unusual way. From my experience most "pure" statisticians don't care about concrete data at all, they are more interested in creating estimators and proving properties about these estimators. Applied statisticians (of which machine learning practitioners is one type) then use these estimators on concrete datasets in order to answer practical questions.

3. "Pure" statistician come up with methods and then test how well those methods behave under various contexts. Applied statisticians then use well-behaved estimators that align with the properties of their data. Machine learning practitioners do the same, they use well-tested concepts like cross-validation (itself a product of statistics) to estimate the accuracy of their models.

4. p-value is a measure of uncertainty, not accuracy. The concept itself is best understood as a form of enhanced induction, where you test a theory about the real world by only having observed a few facts. For example p-value can be applied to check how certain we are about our measure of accuracy on the test set.

5. Accuracy cannot be a substitute for a p-value. Consider a scenario where we ask if there is a difference between two classes, and the overlap of those classes is 99%. Whatever you do with pure prediction you will only be able to get an accuracy of 51%. But with a big enough sample size you will reach an arbitrary small p-value, stating that those two classes are indeed different.

6. Contrary to the statement that statisticians don't care about model generalisation - it's the opposite. Statisticians care about the generalisation of everything, not just accuracy. That's what things like confidence intervals and p-values try to achieve - to give a hint how well some estimate on a sample generalises to a broader population.

7. In my personal opinion one of the bigger differences between the communities of statistics and machine learning (with exceptions of course) is the overall context of the effort. Statisticians place more emphasis on assumptions and selecting the best tool/model for the job at hand. You understand your data, check some assumptions, have a question, and device the best strategy of answering that single question and nothing else. While machine learning people place emphasis on the best overall possible method. For example, I would bet that a lot ML practitioners have a hope of achieving the perfect model that can mimic the human brain and can learn everything and solve multiple problems without being pre-trained, etc. and seems like that is what a big chunk of the community is working towards.