Why does machine learning have metrics such as accuracy, precision or recall to prove best models, but statistics uses hypothesis tests?

Question

I am taking some statistics and machine learning courses and I realized that when doing some model comparison, statistics uses hypothesis tests, and machine learning uses metrics. So, I was wondering, why is that?

Answer 1

As a matter of principle, there is not necessarily any tension between hypothesis testing and machine learning. As an example, if you train 2 models, it's perfectly reasonable to ask whether the models have the same or different accuracy (or another statistic of interest), and perform a hypothesis test.

But as a matter of practice, researchers do not always do this. I can only speculate about the reasons, but I imagine that there are several, non-exclusive reasons:

• The scale of data collection is so large that the variance of the statistic is very small. Two models with near-identical scores would be detected as "statistically different," even though the magnitude of that difference is unimportant for its practical operation. In a slightly different scenario, knowing with statistical certainty that Model A is 0.001% more accurate than Model B is simply trivia if the cost to deploy Model A is larger than the marginal return implied by the improved accuracy.
• The models are expensive to train. Depending on what quantity is to be statistically tested and how, this might require retraining a model, so this test could be prohibitive. For instance, cross-validation involves retraining the same model, typically 3 to 10 times. Doing this for a model that costs millions of dollars to train once may make cross-validation infeasible.
• The more relevant questions about the generalization of machine learning models are not really about the results of repeating the modeling process in the controlled settings of a laboratory, where data collection and model interpretation are carried out by experts. Many of the more concerning failures of ML arise from deployment of machine learning models in uncontrolled environments, where the data might be collected in a different manner, the model is applied outside of its intended scope, or users are able to craft malicious inputs to obtain specific results.
• The researchers simply don't know how to do statistical hypothesis testing for their models or statistics of interest.

Answer 2

This is generally because the use-case, at least historically, for hypothesis testing in statistics is often about simply making a generalization. The use-case in machine learning is to build a useful model, usually under the assumption of the corresponding generalization.

Take, for example, Fisher's Iris Flower Dataset. One question someone might ask is "Do setosa, virginica and versicolor have, on average, different sepal lengths?" We can tackle this question with the scientific method:

1. Hypothesize that they have the same sepal length. (bc this is falsifiable)
2. Attempt to gather evidence to the contrary.
3. Is evidence strong? If so, discard same-sepal-length hypothesis.

The $p$-value in hypothesis testing tries to help answer the "Is evidence strong?" question in this procedure.

In an ML setting, it is generally assumed that these species differ by, for example, sepal length. A question to ask might be "Can we predict {setosa, virginica, versicolor} from the sepal length?" A model is built and its ability to predict the species from the sepal length is measured in precision, recall, accuracy, etc etc etc. Note that since there are many possible models, the precision, recall and accuracy may or may not give you information about whether or not there is a relationship there in the first place. So, for example, we build a decision tree to distinguish these species based on sepal length, and it reports a 33% accuracy (i.e. no better than guessing) -- Does this mean that there is not a relationship, or just that you chose the wrong model?

Of course, in a sense, the hypothesis testing procedure also involves building and evaluating a model. However, the model usually isn't even used explicitly: it merely informs the particular equations that we use to get at "Do the species differ by sepal length?"

Answer 3

Anything can be seen as a "metric", and both groups, statisticians and machine-learners, use plenty of those: accuracy, mean value, estimated parameter of a model, etc. Hypothesis testing is done on top of these "metrics" in order to measure their uncertainty.

For example, if you have 5 male and 5 female students you can measure their heights and get a "metric" for the average height difference between males and females. But the number you get will not reflect the real average difference between all males and all females. Hypothesis testing tries to check if a hypothesis about a population is consistent with the observed "metric".

Same holds for accuracy measurements in machine learning. You build a model and, using a test set of, say, 100 samples, you get an accuracy of 88%. But this is just a measure of accuracy on 100 samples, and not the true accuracy. If you used another 100 samples you would get a slightly different number. So given this accuracy on a set of 100 samples - what can we say about the true accuracy of this classifier? This is where hypothesis testing comes in. And it allows us to answer questions like "how surprising would it be to get an accuracy of 88% on my 100 samples, if the true accuracy of a model is 75%".