How to specify the null hypothesis in hypothesis testing

Question

What is a good rule of thumb for how to choose the question for the null hypothesis. For instance, if I want to check if the hypothesis B is true, should I use B as the null, B as the alternative hypothesis, or NOT B as the null? I hope the question is clear. I know that it has something to do with the error I want to minimize (Type I?), but I keep forgetting how it goes, because I don't have a clear intuition built for it. Thanks.

Answer 1

A rule of the thumb from a good advisor of mine was to set the Null-Hypothesis to the outcome you do not want to be true i.e. the outcome whose direct opposite you want to show.

Basic example: Suppose you have developed a new medical treatment and you want to show that it is indeed better than placebo. So you set Null-Hypothesis $H_0:=$new treament is equal or worse than placebo and Alternative Hypothesis $H_1:=$new treatment is better than placebo.

This because in the course of a statistical test you either reject the Null-Hypothesis (and favor the Alternative Hypothesis) or you cannot reject it. Since your "goal" is to reject the Null-Hypothesis you set it to the outcome you do not want to be true.

Side Note: I am aware that one should not set up a statistical test to twist it and break it until the Null-Hypothesis is rejected, the casual language was only used to make this rule easier to remember.

This also may be helpful: What is the meaning of p values and t values in statistical tests? and/or What is a good introduction to statistical hypothesis testing for computer scientists?

Answer 2

If hypothesis B is the interesting hypothesis you can take not-B as the null hypothesis and control, under the null, the probability of the type I error for wrongly rejecting not-B at level $\alpha$. Rejecting not-B is then interpreted as evidence in favor of B because we control the type I error, hence it is unlikely that not-B is true. Confused ... ?

Take the example of treatment vs. no treatment in two groups from a population. The interesting hypothesis is that treatment has an effect, that is, there is a difference between the treated group and the untreated group due to the treatment. The null hypothesis is that there is no difference, and we control the probability of wrongly rejecting this hypothesis. Thus we control the probability of wrongly concluding that there is a treatment effect when there is no treatment effect. The type II error is the probability of wrongly accepting the null when there is a treatment effect.

The formulation above is based on the Neyman-Pearson framework for statistical testing, where statistical testing is seen as a decision problem between to cases, the null and the alternative. The level $\alpha$ is the fraction of times we make a type I error if we (independently) repeat the test. In this framework there is really not any formal distinction between the null and the alternative. If we interchange the null and the alternative, we interchange the probability of type I and type II errors. We did not, however, control the type II error probability above (it depends upon how big the treatment effect is), and due to this asymmetry, we may prefer to say that we fail to reject the null hypothesis (instead of that we accept the null hypothesis). Thus we should be careful about concluding that the null hypothesis is true just because we can't reject it.

In a Fisherian significance testing framework there is really only a null hypothesis and one computes, under the null, a $p$-value for the observed data. Smaller $p$-values are interpreted as stronger evidence against the null. Here the null hypothesis is definitely not-B (no effect of treatment) and the $p$-value is interpreted as the amount of evidence against the null. With a small $p$-value we can confidently reject the null, that there is no treatment effect, and conclude that there is a treatment effect. In this framework we can only reject or not reject (never accept) the null, and it is all about falsifying the null. Note that the $p$-value does not need to be justified by an (imaginary) repeated number of decisions.

Neither framework is without problems, and the terminology is often mixed up. I can recommend the book Statistical evidence: a likelihood paradigm by Richard M. Royall for a clear treatment of the different concepts.

Answer 3

The "frequentist" response is to invent a null hypothesis of the form "not B" and then argue against "not B", as in Steffen's response. This is the logical equivalent of making the argument "You are wrong, therefore I must be right". This is the kind of reasoning politician's use (i.e. the other party is bad, therefore we are good). It is quite difficult to deal with more than 1 alternative under this sort of reasoning. This is because that "you are wrong, therefore I am right" argument only makes sense when it is not possible for both to be wrong, which can certainly happen when there is more than one alternative hypothesis.

The "Bayesian" response is to simply calculate the probability of the hypothesis that you are interested in testing, conditional on whatever evidence you have. Always this contains prior information, which is simply the assumptions you have made to made your problem well posed (all statistical procedures rely on prior information, Bayesian ones just make them more explicit). It also usually consists of some data, and we have by bayes theorem

$$P(H_{0}|DI)=\frac{P(H_{0}|I)P(D|H_{0}I)}{\sum_{k}P(H_{k}|I)P(D|H_{k}I)}$$

This form is independent of what is called the "null" and what is called the "alternative", because you have to calculate exactly the same quantities for every hypothesis that you are going to consider - the prior and the likelihood. This is in a sense, analogous to calculate the "type 1" and "type 2" error rates in Neyman Pearson hypothesis testing, simply because a "type 2" error rate when $H_0$ is the "null" is the same thing as the "type 1" error rate with $H_0$ is the "alternative". It is only the connotations implied by the words "null" and "alternative" which make them seem different. You can show equivalence in the case of the "Neyman Pearson Lemma" when there are two hypothesis, for this is simply the likelihood ratio, which is given at once by taking the odds of the above bayes theorem:

$$\frac{P(H_{0}|DI)}{P(H_{1}|DI)}=\frac{P(H_{0}|I)}{P(H_{1}|I)}\times\frac{P(D|H_{0}I)}{P(D|H_{1}I)}=\frac{P(H_{0}|I)}{P(H_{1}|I)}\times\Lambda$$

So the decision problems are the same: accept $H_0$ when $\Lambda > \tilde{\Lambda}$ for some cut-off $\tilde{\Lambda}$, and accept $H_1$ otherwise. Thus, the procedures are basically different rationales for choosing the cut-off value, or decision boundary. "Bayesians" would say it should be the product of the prior odds times the loss ratio $\frac{L_2}{L_1}$ where $L_1$ is the "type 1 error loss" and $L_2$ is the "type 2 error loss". These are losses, not probabilities, which describe the relative severity of making each of the two errors. The frequentist criterion is to minimise the one of the average error rates, type 1 or 2, while keeping the other fixed. But because they lead to the same form of decision boundary, we can always find an equivalent bayesian prior*loss ratio for every frequentist minimised error rate.

In short, if you are using the likelihood ratio to test your hypothesis, it does not matter what you call the null hypothesis. Switching the null to the alternative just changes the decision to $\Lambda^{-1}<\tilde{\Lambda}^{-1}$ which is mathematically the same thing (you will make the same decision - but based on inverse chi-square cut-off rather than chi-square for your p-value). Playing word games with "failing to reject the null" just doesn't apply to the hypothesis test, because it is a decision, so if there are only two options, then "failing to reject the null" means the same thing as "accepting the null".

Answer 4

The null hypothesis should generally assume that differences in a response variable are due to error alone.

For example if you want to test the effect of some factor A on response x, then the null would be: $H_0$ = There is no effect of A on response x.

Failing to reject this null hypothesis would be interpreted as:

1) any differences in x are due to error alone and not A or,

2) that the data are inadequate to detect a difference even though one exists (see Type 2 error below).

Rejecting this null hypothesis would be interpreted as the alternative hypothesis: $H_a$ = There is an effect of A on response x, is true.

Type 1 and Type 2 errors are related to the use of the null hypothesis but not its designation really. Type 1 error occurs when you reject $H_0$ even though it is true - that is, you incorrectly conclude an effect of A on x when one didn't exist. Type 2 error occurs when you fail to reject the $H_0$ even though it is false - that is, you incorrectly conclude no effect of A on x even though one exists.