Why is Binary Classification not a Hypothesis Test?

Question

I'm having a hard time finding the right way to explain to an engineer friend why binary classification isn't quite the same as a statistical hypothesis test. Clearly, in both cases we are choosing between two alternatives using some statistical procedure. Yet the intent, the language, and the methodology are rather different. How would you explain the difference?

Answer 1

It seems perfectly reasonable to me that the binary classification of a given pattern could be analogous to a hypothesis test, but it isn't necessarily. It also depends on exactly what situation you have in mind here. Let us assume that you have a classifier (hopefully a good one), and want to use it determine which class a given pattern belongs to. This should be a new pattern for which you don't already know the true class. It is common in machine learning to assess the value of your classifier by comparing its predicted classes to known (true) classes, but that is a different endeavor. Then,

• From within the Neyman-Pearson approach to hypothesis testing (cf., here), you will act as though the null is correct unless there is sufficient evidence to reject it. To be clear, that does not mean that you have 'proven' the null to be true (cf., here), you will eventually make errors in both directions. The key to this is deciding what long-run error rates you think you can live with. Typically, the null and alternative are not treated symmetrically—preference is given to the null. Thus for example, people will usually only reject the null if the evidence is sufficient that their long-run 'type I' error rate is 5%. Generally, a study to test the hypothesis is constructed such that the null will be rejected 80% of the time when the alternative obtains. Those facts imply that people are more comfortable erring on the side of not rejecting the null, than the other way around. But that is a particular value judgment regarding the demerits of the different types of mistakes, there is nothing logically necessary about that.
• On the other hand, when classifying a novel pattern in machine learning, it is typical that all patterns are classified, and are classified as the maximum a-posteriori class. That is, a pattern will be classified as class A if the classifier suggests it is more likely to be an A than a not-A. This again is a value judgment. Classifiers can be 'weighted' so that they will prioritize sensitivity or specificity.

Thus, these represent different cultural and conceptual frameworks, but can be put in correspondence with regard to the underlying logical structure of the two activities.

---

There are other perspectives we could take on comparing and contrasting these as well. For example, we could discuss whether the classifier performs adequately (judged according to some criterion) based on how well the function the classifier embodies, $\hat f({\rm data})$ mimics the true underlying function $f({\rm data})$, and whether / how closely the assumptions of the particular hypothesis test are met. Similarly, we could contrast how classifiers are trained in machine learning (e.g., by minimizing cross validation error), vs. how models are built to create a context within which a specific hypothesis can be tested.

For a broader viewpoint, you may be interested in my answer here: What is the difference between data mining, statistics, machine learning and AI?

Answer 2

The two are not necessarily non-overlapping in practice. I would also note that hypothesis-testing and statistical-significance seem slightly different to me (at least going by the tag descriptions on this site): Not all hypothesis testing has to be comparing to a "null" (or "random/by chance/no effect") alternative.

That said, hypothesis tests are perhaps typically associated with pre-defined hypotheses, commonly specified in terms of parametric distributions.

So for example, in training a gaussian-mixture model, the parameters ($\mu_k$,$\Sigma_k$) and data labels $k_i\in\mathrm{components}$ (where $i\in\mathrm{data}$) typically vary. However for a fixed set of component parameters, deciding which component (class) a point belongs to is at least akin to hypothesis testing (e.g. when classifying new examples, or deciding labels in the "E-step" of E-M training).

So I guess the key distinction (in my view), would be that in (classical?) statistical hypothesis testing, changing the hypothesis is strictly not allowed, once the data have been seen.

(See, for example, the controversial phenomenon of "p-hacking".)