Why we reject the null hypothesis at the 0.05 level and not the 0.5 level (as we do in the Classification)

Question

Hypothesis testing is akin to a Classification problem. So say, we have 2 possible labels for an observation (subject) -- Guilty vs. Non-Guilty. Let Non-Guilty be the null Hypothesis. If we viewed the problem from a Classification viewpoint we would train a Classifier which would predict the probability of the subject belonging in each of the 2 Classes, given the Data. We would then pick the Class with the highest probability. In that case 0.5 probability would be the natural threshold. We might vary the threshold in case we assigned different costs to False Positive vs. False Negative errors. But rarely we would go so extreme as setting the threshold at 0.05, i.e. assign the subject to Class "Guilty" only if the probability is 0.95 or higher. But if I understand well, this is what we are doing as a standard practice when we view the same problem as a problem of Hypothesis testing. In this latter case, we will not assign the label "Non-Guilty" --equivalent to assigning the label "Guilty"-- only if the probability of being "Non-Guilty" is less than 5%. And perhaps this might make sense if we truly want to avoid to convict innocent people. But why this rule should prevail in all Domains and all cases?

Deciding which Hypothesis to adopt is equivalent to defining an Estimator of the Truth given the Data. In Maximum Likelihood Estimation we accept the Hypothesis that is more likely given the Data -- not necessarily though overwhelmingly more likely. See the graph below:

Using a Maximum Likelihood approach we would favor the Alternative Hypothesis in this example if the value of the Predictor was above 3, e.g. 4, although the probability of this value to have been derived from the Null Hypothesis would have been larger than 0.05.

And while the example with which I begun the post is perhaps emotionally charged, we could think of other cases, e.g. a technical improvement. Why we should give such an advantage to the Status Quo when the Data tell us that the probability that the new solution is an improvement is greater than the probability that it is not?

Answer 1

Say you end up in court and you did not do it. Do you think it is fair that you still have a 50% chance of being found guilty? Is a 50% chance of being innocent "guilty beyond reasonable doubt"? Would you think it is fair that you had a 5% chance of being found guilty even though you did not do it? If I were in court I would consider 5% not conservative enough.

You are right that the 5% is arbitrary. We could just as well choose 2%, or 1%, or if you are nerdy $\pi$% or $e$%. There are people who are willing to accept 10%, but 50% will never be acceptable.

---

In response to your edit of the question:

Your idea would be reasonable if all hypotheses were created equal. However, that is not the case. We typically care about the alternative hypothesis, so we strengthen our argument if we choose a low $\alpha$. In that sense, the example you chose originally illustrates that point well.

Answer 2

It is like you say - it depends on how important False Positive and False Negative errors are.

In the example you use, as Maarten Buis already answered, being convicted if there is a 50% chance that you were innocent is hardly fair.

When applying it to research, look at this way: Imagine you want to know if a certain new medication helps against a certain disease. Say that you find a difference between your treatment group and your control group in favour of the treatment. Great! The medicine must work, right? You can reject the null hypothesis that the medication does not work. Your p-value is 0.49! There is a higher chance that the effect you found was based on the truth rather than by chance!
Now consider this: the medication has nasty adverse effects. You only want to take it if you're convinced it works. And are you? No, because there is still a 51% chance that the difference you found between the two groups was purely by chance.

I can imagine that there are domains where you're satisfied with e.g. 10%. I've seen articles where 10% is accepted. I've also seen articles where they chose 2%. It depends on how important you think it is that you're convinced that rejecting the null hypothesis will be based on the truth and not on chance. I can hardly imagine a situation where you're satisfied with a 50% chance that the difference you found was based on pure luck.

Answer 3

Other answers have pointed out that it all depends on how you relatively value the different possible errors, and that in a scientific context $.05$ is potentially quite reasonable, an even more stringent criterion is also potentially quite reasonable, but that $.50$ is unlikely to be reasonable. That is all true, but let me take this in a different direction and challenge the assumption that lies behind the question.

---

You take "[h]ypothesis testing [to be] akin to a Classification problem". The apparent similarity here is only superficial; that isn't really true in a meaningful sense.

In a binary classification problem, there really are just two classes; that can be established absolutely and a-priori. Hypothesis testing isn't like that. Your figure displays a null and an alternative hypothesis as they are often drawn to illustrate a power analysis or the logic of hypothesis testing in a Stats 101 class. The figure implies that there is one null hypothesis and one alternative hypothesis. While it is (usually) true that there only one null, the alternative isn't fixed to be only a single point value of the (say) mean difference. When planning a study, researchers will often select a minimum value they want to be able to detect. Let's say that in some particular study it is a mean shift of $.67$ SDs. So they design and power their study accordingly. Now imagine that the result is significant, but $.67$ does not appear to be a likely value. Well, they don't just walk away! The researchers would nonetheless conclude that the treatment makes a difference, but adjust their belief about the magnitude of the effect according to their interpretation of the results. If there are multiple studies, a meta-analysis will help refine the true effect as data accumulates. In other words, the alternative that is proffered during study planning (and that is drawn in your figure) isn't really a singular alternative such that the researchers must choose between it and the null as their only options.

Let's go about this a different way. You could say that it's quite simple: either the null hypothesis is true or it is false, so there really are just two possibilities. However, the null is typically a point value (viz., $0$) and the null being false simply means that any value other than exactly $0$ is the true value. If we recall that a point has no width, essentially $100\%$ of the number line corresponds to the alternative being true. Thus, unless your observed result is $0.\bar{0}$ (i.e., zero to infinite decimal places), your result will be closer to some non-$0$ value than it is to $0$ (i.e., $p<.5$). As a result, you would always end up concluding the null hypothesis is false. To make this explicit, the mistaken premise in your question is that there is a single, meaningful blue line (as depicted in your figure) that can be used as you suggest.

The above need not always be the case however. It does sometimes occur that there are two theories making different predictions about a phenomenon where the theories are sufficiently well mathematized to yield precise point estimates and likely sampling distributions. Then, a critical experiment can be devised to differentiate between them. In such a case, neither theory needs to be taken as the null and the likelihood ratio can be taken as the weight of evidence favoring one or the other theory. That usage would be analogous to taking $.50$ as your alpha. There is no theoretical reason this scenario couldn't be the most common one in science, it just happens that it is very rare for there to be two such theories in most fields right now.

Answer 4

To add to the very good previous answers: Yes, 5% is arbitrary, but regardless of the specific threshold you pick, it has to be reasonably small, otherwise hypothesis testing makes little sense.

You're looking for an effect and want to make sure your results are not purely due to chance. To that extent, you set a significance level which says basically "If there were actually no effect (null hypothesis is true), this would be the probability to still get such results (or more extreme) by pure chance". Setting this too high will result in lots of false positives, and undermine your ability to get a meaningful answer to your research question.

As always, there's a trade-off involved, so the research community came up with this 5% guideline. But it's different in different fields. In particle physics, it's more like 0.00001% or something.

Answer 5

Classification and hypothesis testing are different and been used differently. In most cases, people use

• "Classification"" to perform the task of "classifying something according to shared qualities or characteristics".
• And use "hypothesis testing" to verify some "significant discoveries".

Note that, in hypothesis testing, the "null hypothesis" is "common sense", but if we can reject null hypotheses then we have a break though.

This is why we have a more strict criteria in hypothesis testing. Think example of developing new drags, we want to be very careful to say the is significant and effective.