5
$\begingroup$

I'm having a hard time finding the right way to explain to an engineer friend why binary classification isn't quite the same as a statistical hypothesis test. Clearly, in both cases we are choosing between two alternatives using some statistical procedure. Yet the intent, the language, and the methodology are rather different. How would you explain the difference?

| cite | improve this question | | | | |
$\endgroup$
6
$\begingroup$

It seems perfectly reasonable to me that the binary classification of a given pattern could be analogous to a hypothesis test, but it isn't necessarily. It also depends on exactly what situation you have in mind here. Let us assume that you have a classifier (hopefully a good one), and want to use it determine which class a given pattern belongs to. This should be a new pattern for which you don't already know the true class. It is common in machine learning to assess the value of your classifier by comparing its predicted classes to known (true) classes, but that is a different endeavor. Then,

  • From within the Neyman-Pearson approach to hypothesis testing (cf., here), you will act as though the null is correct unless there is sufficient evidence to reject it. To be clear, that does not mean that you have 'proven' the null to be true (cf., here), you will eventually make errors in both directions. The key to this is deciding what long-run error rates you think you can live with. Typically, the null and alternative are not treated symmetrically—preference is given to the null. Thus for example, people will usually only reject the null if the evidence is sufficient that their long-run 'type I' error rate is 5%. Generally, a study to test the hypothesis is constructed such that the null will be rejected 80% of the time when the alternative obtains. Those facts imply that people are more comfortable erring on the side of not rejecting the null, than the other way around. But that is a particular value judgment regarding the demerits of the different types of mistakes, there is nothing logically necessary about that.
  • On the other hand, when classifying a novel pattern in machine learning, it is typical that all patterns are classified, and are classified as the maximum a-posteriori class. That is, a pattern will be classified as class A if the classifier suggests it is more likely to be an A than a not-A. This again is a value judgment. Classifiers can be 'weighted' so that they will prioritize sensitivity or specificity.

Thus, these represent different cultural and conceptual frameworks, but can be put in correspondence with regard to the underlying logical structure of the two activities.


There are other perspectives we could take on comparing and contrasting these as well. For example, we could discuss whether the classifier performs adequately (judged according to some criterion) based on how well the function the classifier embodies, $\hat f({\rm data})$ mimics the true underlying function $f({\rm data})$, and whether / how closely the assumptions of the particular hypothesis test are met. Similarly, we could contrast how classifiers are trained in machine learning (e.g., by minimizing cross validation error), vs. how models are built to create a context within which a specific hypothesis can be tested.

For a broader viewpoint, you may be interested in my answer here: What is the difference between data mining, statistics, machine learning and AI?

| cite | improve this answer | | | | |
$\endgroup$
  • $\begingroup$ (+1) This helps to clarify things. Without going so far as to open a new question (yet), I wonder how this sense of "hypothesis test" compares to approaches used in autonomous vehicles/robotics? More specifically, the "online hypothesis generation/testing loop" of particle filters uses for things like SLAM. For example "hypothesis" as used here. $\endgroup$ – GeoMatt22 Oct 14 '16 at 16:39
  • $\begingroup$ This was very helpful indeed. The conceptual context I'm working in is medical diagnosis, which tends to favor the "classification" framework over the hypothesis testing framework (I'm sure there is historical motivation), but again we choose a method with a certain long-run error metric in mind (typically ROC based, not just type-i or type-ii error). So diagnosis could be couched as a hypothesis test where "null" is say "signal present". Type I error then corresponds to the false negative rate, which we almost always want to minimize. $\endgroup$ – icurays1 Oct 14 '16 at 16:51
  • $\begingroup$ Those topics are really outside my familiarity, @GeoMatt22. I'm afraid I just don't really know. It does bring to mind Cardinal's answer here. $\endgroup$ – gung - Reinstate Monica Oct 14 '16 at 19:09
  • $\begingroup$ You're welcome, @icurays1. If you want a classifier to say 'yes' unless the case for 'no' is really strong, it means you are prioritizing sensitivity over specificity, from a ml / ROC-esque perspective. I don't have a problem with saying that that activity is analogous to hypothesis testing, although they many not look quite the same on the surface. The thing about a logical structure is that instances can occur in any number of contexts, even where things look very different. $\endgroup$ – gung - Reinstate Monica Oct 14 '16 at 19:15
  • $\begingroup$ @gung thanks for the pointer. I note that in the comments of that question (and apparently its downvote history) there is some controversy over whether the examples count as "hypothesis tests". $\endgroup$ – GeoMatt22 Oct 14 '16 at 19:30
1
$\begingroup$

The two are not necessarily non-overlapping in practice. I would also note that and seem slightly different to me (at least going by the tag descriptions on this site): Not all hypothesis testing has to be comparing to a "null" (or "random/by chance/no effect") alternative.

That said, hypothesis tests are perhaps typically associated with pre-defined hypotheses, commonly specified in terms of distributions.

So for example, in training a model, the parameters ($\mu_k$,$\Sigma_k$) and data labels $k_i\in\mathrm{components}$ (where $i\in\mathrm{data}$) typically vary. However for a fixed set of component parameters, deciding which component (class) a point belongs to is at least akin to hypothesis testing (e.g. when classifying new examples, or deciding labels in the "E-step" of E-M training).

So I guess the key distinction (in my view), would be that in (classical?) statistical hypothesis testing, changing the hypothesis is strictly not allowed, once the data have been seen.

(See, for example, the controversial phenomenon of "p-hacking".)

| cite | improve this answer | | | | |
$\endgroup$
  • $\begingroup$ "...changing the hypothesis is not strictly allowed..." I think that's the key idea I've been trying to articulate - a hypothesis is usually stated before data is collected, typically in terms of a parametric model. The data is then tested against the model to make a decision, with a statistical significance. I suppose this process could be iterated in the sense that the model could then be updated to serve as the "next hypothesis". $\endgroup$ – icurays1 Oct 14 '16 at 5:09
  • $\begingroup$ OK. I edited slightly to emphasize this point visually. (Perhaps I went overboard with the tag-links?) $\endgroup$ – GeoMatt22 Oct 14 '16 at 5:12
  • 1
    $\begingroup$ I'm not sure about this. How can you have a hypothesis test w/o a null, eg? Do you mean that the null doesn't have to be 0? IMO, hypothesis test should be pre-defined, but it certainly doesn't always play out that way in practice, & HT need not pertain to some named parametric distribution (eg, that isn't the way I would think of the Mann-Whitney U-test). $\endgroup$ – gung - Reinstate Monica Oct 14 '16 at 15:14
  • 1
    $\begingroup$ @gung I added a qualification on "hypotheses are commonly parametric" (I thought I had written that originally ... it was past my bedtime though). I do not know about your "null" point, which may come down to semantics (in which case I would defer to you, as I am not a statistician). I was thinking something like "null vs. alternative hypothesis = no effect vs. effect" contrasted with "alternative hypotheses = effect 1 vs. effect 2 vs ...". I had vaguely inferred this from the wiki's for statistical-significance vs. hypothesis-testing. $\endgroup$ – GeoMatt22 Oct 14 '16 at 15:25
  • $\begingroup$ "Commonly parametric" is probably OK (albeit vague). Your idea that you could test "effect 1 vs effect 2" seems to imply a situation in which you have a null & an alternative hypothesis, but that the null isn't necessarily 0 (although it's slightly ambiguous). So I would say that's OK, too, as far as my opinion goes; remember that my background is ancient Chinese philosophy, not statistics. $\endgroup$ – gung - Reinstate Monica Oct 14 '16 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.