21
$\begingroup$

I am taking some statistics and machine learning courses and I realized that when doing some model comparison, statistics uses hypothesis tests, and machine learning uses metrics. So, I was wondering, why is that?

$\endgroup$
6
  • 9
    $\begingroup$ Statisticians can use metrics as well, several of these were invented by statisticians rather than machine learners. Also many statisticians oppose the (over-)use of significance tests. $\endgroup$ Commented Oct 29, 2021 at 21:56
  • 5
    $\begingroup$ Note that accuracy, precision, recall etc. are intuitive - but also highly misleading. Every criticism of accuracy at the following thread applies equally to all other evaluation measures of "hard" 0-1 classifications: Why is accuracy not the best measure for assessing classification models? Instead, one should use probabilistic classifications and evaluate them using proper scoring rules. $\endgroup$ Commented Oct 30, 2021 at 7:01
  • $\begingroup$ These "metrics" or measures are among many possible similarity coefficients for binary or multinomial data. They, in s sense, can be seen as measures of correlation. Like almost any coefficient, they can be tested for null hypothesis of zero value, giving a p-value. So, your question boils down to sacramental "effect size or significance level?" $\endgroup$
    – ttnphns
    Commented Oct 31, 2021 at 14:36
  • $\begingroup$ @ChristianHennig and machine learning people do perform statistical tests on their metrics (at least they ought to, even if they don't necessarily mean very much). $\endgroup$ Commented Oct 31, 2021 at 17:33
  • 1
    $\begingroup$ Note on the thread that @StephanKolassa mentions, there are examples where accuracy is the primary quantity of interest in the practical application and an example of how proper scoring rules can give the wrong model choice in such applications. It is a gross overstatement to say that accuracy is highly misleading - it has it's advantages and disadvantages, just like everything else. Part of it depends on whether you are interested in model choice or model evaluation - they are not necessarily the same thing. $\endgroup$ Commented Oct 31, 2021 at 17:37

3 Answers 3

21
$\begingroup$

As a matter of principle, there is not necessarily any tension between hypothesis testing and machine learning. As an example, if you train 2 models, it's perfectly reasonable to ask whether the models have the same or different accuracy (or another statistic of interest), and perform a hypothesis test.

But as a matter of practice, researchers do not always do this. I can only speculate about the reasons, but I imagine that there are several, non-exclusive reasons:

  • The scale of data collection is so large that the variance of the statistic is very small. Two models with near-identical scores would be detected as "statistically different," even though the magnitude of that difference is unimportant for its practical operation. In a slightly different scenario, knowing with statistical certainty that Model A is 0.001% more accurate than Model B is simply trivia if the cost to deploy Model A is larger than the marginal return implied by the improved accuracy.
  • The models are expensive to train. Depending on what quantity is to be statistically tested and how, this might require retraining a model, so this test could be prohibitive. For instance, cross-validation involves retraining the same model, typically 3 to 10 times. Doing this for a model that costs millions of dollars to train once may make cross-validation infeasible.
  • The more relevant questions about the generalization of machine learning models are not really about the results of repeating the modeling process in the controlled settings of a laboratory, where data collection and model interpretation are carried out by experts. Many of the more concerning failures of ML arise from deployment of machine learning models in uncontrolled environments, where the data might be collected in a different manner, the model is applied outside of its intended scope, or users are able to craft malicious inputs to obtain specific results.
  • The researchers simply don't know how to do statistical hypothesis testing for their models or statistics of interest.
$\endgroup$
1
  • $\begingroup$ (+1) Another issue is that if you are developing a new machine learning algorithm and compare it to the state of the art, you would not expect it to be statistically superior, at least at the 95% level of significance. When writing methods papers, we don't really want to stifle the promulgation of promising ideas by being so stringent (I think a 90% level would be fine). For many benchmark problems an SVM is not statistically superior to a regularised radial basis function when both are fitted according to best practice. $\endgroup$ Commented Nov 2, 2021 at 19:59
6
$\begingroup$

This is generally because the use-case, at least historically, for hypothesis testing in statistics is often about simply making a generalization. The use-case in machine learning is to build a useful model, usually under the assumption of the corresponding generalization.

Take, for example, Fisher's Iris Flower Dataset. One question someone might ask is "Do setosa, virginica and versicolor have, on average, different sepal lengths?" We can tackle this question with the scientific method:

  1. Hypothesize that they have the same sepal length. (bc this is falsifiable)
  2. Attempt to gather evidence to the contrary.
  3. Is evidence strong? If so, discard same-sepal-length hypothesis.

The $p$-value in hypothesis testing tries to help answer the "Is evidence strong?" question in this procedure.

In an ML setting, it is generally assumed that these species differ by, for example, sepal length. A question to ask might be "Can we predict {setosa, virginica, versicolor} from the sepal length?" A model is built and its ability to predict the species from the sepal length is measured in precision, recall, accuracy, etc etc etc. Note that since there are many possible models, the precision, recall and accuracy may or may not give you information about whether or not there is a relationship there in the first place. So, for example, we build a decision tree to distinguish these species based on sepal length, and it reports a 33% accuracy (i.e. no better than guessing) -- Does this mean that there is not a relationship, or just that you chose the wrong model?

Of course, in a sense, the hypothesis testing procedure also involves building and evaluating a model. However, the model usually isn't even used explicitly: it merely informs the particular equations that we use to get at "Do the species differ by sepal length?"

$\endgroup$
4
$\begingroup$

Anything can be seen as a "metric", and both groups, statisticians and machine-learners, use plenty of those: accuracy, mean value, estimated parameter of a model, etc. Hypothesis testing is done on top of these "metrics" in order to measure their uncertainty.

For example, if you have 5 male and 5 female students you can measure their heights and get a "metric" for the average height difference between males and females. But the number you get will not reflect the real average difference between all males and all females. Hypothesis testing tries to check if a hypothesis about a population is consistent with the observed "metric".

Same holds for accuracy measurements in machine learning. You build a model and, using a test set of, say, 100 samples, you get an accuracy of 88%. But this is just a measure of accuracy on 100 samples, and not the true accuracy. If you used another 100 samples you would get a slightly different number. So given this accuracy on a set of 100 samples - what can we say about the true accuracy of this classifier? This is where hypothesis testing comes in. And it allows us to answer questions like "how surprising would it be to get an accuracy of 88% on my 100 samples, if the true accuracy of a model is 75%".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.