Accepting the alternative hypothesis when it is not the complement of the null

Question

When looking at the hypothesis testing literature, I have noticed that there are different ways of phrasing your conclusions after having done a test.

When the test fails to reject, there seems to be strong consensus that we cannot say that we "accept the null" and there are very good reasons for this. This mainly stems from the idea that hypothesis tests are basically "proofs by contradiction given a particular significance level". If we fail to reject under the null (i.e. the data is not very "extreme" given the null, it still does not mean the null is true).

When the test rejects, we usually say something like "on an $\alpha$% significance level, we can say that $H_0$ is not true". My question is on what we can subsequently conclude about $H_1$. I know that all of these conclusions are not "proofs" and only hold under a particular significance level $\alpha$ which follows from restricting the Type I error of the test. I distinguish two cases:

1. $H_1$ is the complement of $H_0$ (this of course depends on the underlying parameter space). One example of this is the classical case where we test $H_0: \theta =0 $ vs $H_1: \theta \neq 0.$ I think that most people would say that when your test rejects here, you can say that $H_0$ is not true and thus $H_1$ has to be true. Note that the latter is not a very strong claim for the classical example, as this contains almost the whole parameter space. Nonetheless, when the null is also a composite hypothesis, I think we can still say that $H_1$ holds when it is the complement of $H_1$.

2. $H_1$ is not the complement of $H_0$. One classical example of this would be a simple null and simple alternative but the underlying parameter space also contains other values. Here, I would say that we cannot conclude the alternative when we reject, we can only say that the null is not true.

Does anyone have any objections to the above claims or any other thoughts? I think it all comes down to the fact that we are dealing with stochastic data and it is hard to make definitive claims about underlying DGPs. However, for policy recommendations etc., we still would like to make some kind of claim and that is why I am asking this question.

Edit: in the standard NHST framework, the alternative hypothesis is always the complement of the null. Therefore, point (2) is not well formulated. Thanks to all comments which provided useful insights.

Answer 1

There are already many good points made in the comments. I want to add one more thing here, and I do it as an answer so that the question has one:

In short:

Hypothesis tests are best thought of helping deciding how to act, not what to believe - it would be better to say "if significant, we will act as if H0 is not true and H1 is true", rather than "we can say that H0 is not true, so we conclude H1 is true". This is because hypotheses in a frequentist worldview are either true or they are not, so statements implying "true under a 5% significance level" are at best misleading.

---

Elaboration:

However, helping how to act can fit quite nicely into the policy recommendation scenario you pointed out. So let's take this as an example. Let's say you have some data, and you want to know based on that data if a policy should be implemented or not - maybe it was tested in a small sample. Now the intent of a hypothesis test (and I do not say NHST here to indicate what the original spirit of hypothesis testing is, but I do not think it matters here) is helping you to decide what to do by bounding your probability of making the wrong decision. Depending on reality and your data, there are four possibilities:

1. Data indicates policy is helpful, and this is true,
2. data indicates policy is not helpful, and this is true

both are good because when you follow the test, you would make the correct decision

3. data indicates policy is helpful, but this is false (type 1 error),
4. data indicates policy is not helpful but this is false (type 2 error).

Before the study you need to set your error rates α and β in a way that reflects the costs of each error - in my personal opinion, being in control of one's own error rates is one of the main advantages of hypothesis testing, which is unfortunately rarely used (for multiple reasons I suppose). Ideally, when your policy is cheap and easy to implement, you might lean on the side of higher power, while when it is expensive and requires a lot of work, maybe you rather want to be on the safe side and not act as if it works too early.

When you then follow the procedure of a test, your error rates will be bounded by these self-imposed values. For instance, if you always set alpha to be 5%, then you will wrongly reject your null at most 5% of the tests you are doing.

So you are right that it theoretically follows that H1 must be true if H0 isn't, the problem is just that you can never be sure that this is really the case if you cannot find specific counterexamples but just samples with more or less extreme values. But you can come arbitrarily close when setting the error rates correctly.

Answer 2

OK, this answer is triggered as much by @DikranMarsupial's comment as it is by the original question. However, read on, as I will tie it to the questions of null and alternative hypotheses.

Dikran Marsupial wrote:

BTW, there are occasions where we use a null hypothesis that we know a-priori is false, for example a perfectly unbiased coin where p(head) = p(tail) = 0.5. A real world coin is an asymmetric object that will have been imperfectly manufactured, so there is no such thing as a perfectly unbiased coin.

The "coin" is a real world artefact and it may or may not be possible for it to be "perfectly unbiassed", but that is almost entirely irrelevant as the actual tossing and catching of the coin are far more powerful influences on the observed ratio of heads and tails than the balance of design or minting of the coin. However, the unbiassedness the real world coin tossing setup is not the null hypothesis of a hypothesis test (or of a significance test).

Statistical tests live entirely within statistical models. They do not concern coins, or the heights of people, or any other real-world objects or properties. They concern only the values of parameters within the statistical model. That means that the statistical inferences that come from statistical analyses have to be extrapolated to become relevant to the real world things that are the subjects of scientific hypotheses. I've written about this at some length elsewhere.

All of that is to say that Dikran's comment may be true but it not relevant to statistical hypotheses, be they null or alternative.

Now, back to the $H_0$ and $H_1$ of the original question. I am assuming that the question is about Neyman–Pearsonian hypothesis testing even though it says "NHST", which refers to a troublesome hybrid that can be read about here and here.

Neyman–Pearsonian hypothesis testing is a method designed to protect against long run errors and not to decide on the merits of any hypotheses. Neyman and Pearson were quite explicit about that in the original paper. The hypothesis test procedure with its dichotomous 'significant'/'not significant' outcome and the consequent decision 'to act as if' the null is false in case of 'significant' has nothing to say about what might be true in the real world. It does not even say what parameter values within the statistical model might be better or worse approximations of the interesting feature of the real world data generating system.

In my mind those things makes a hypothesis test result a terrible guide for policy recommendations. The p-values of a significance test are not much better in that regard, but at least they are not all-or-none. If you really want to compare the merits of statistical hypotheses on the basis of experimental evidence then the likelihood ratio of the parameter values may be the best possible guide.

Answer 3

tl;dr: If you're comparing two nested models, and you fail to reject the smaller model $H_0$, you don't believe that $H_0$ is true---you just don't have enough data to fit the larger model $H_1$. And if you do reject $H_1$, you don't believe that $H_1$ is true either---you just seem to have enough data to fit the larger model's extra parameters, without fearing that sampling variation has swamped the signal in the data.

But even if your hypothesis test rejects $H_0$, there are many other concerns besides sampling variation that also need to be checked & ruled out... and aren't addressed by a hypothesis test. That's why the simple deductive logic of "if we reject $H_0$, then logically we must accept $H_1$" doesn't describe how we use hypothesis tests.

---

Hypothesis testing is poorly named and isn't a great tool for deciding which of two arbitrary hypotheses to believe / act on. If you want to compare two hypotheses with another and decide which is more plausible, consider the Bayesian or likelihoodist approaches mentioned in other answers/comments.

But hypothesis tests can be reasonably useful for the limited purpose of deciding if you have enough data to fit a larger model, or if you can only afford to fit a smaller model. It answers a question about your study design, not about the population being studied. People complain that "with large enough $n$, you can reject any $H_0$"---but that's not a bug, it's a feature.

Say you have two groups (perhaps treatment & control in an experiment), and your substantive question is about which group has a higher mean (does my new drug lower or raise blood pressure on average?). There's a lot of ways a study could go wrong and lead you to reporting two sample means $\hat\mu_{treatment}$ and $\hat\mu_{control}$ which are in the "wrong" order: the treatment looked better in this sample but it's actually worse in the population, or vice versa.

You want to avoid this, so you go through the long list of things to think about during study design & data analysis:

• You try to prevent confounding from causing this, by using random assignment and double-blinding.
• You try to prevent measurement error from causing this, by using accurate instruments and consistent protocols.
• And so on...
• Among other things, you also try to prevent sampling variation from causing this, preferably by planning a large-enough sample size before data collection, but also by checking it after data collection with a hypothesis test.

We almost never actually believe "$H_0:$ the two groups have identical means" is plausible. So we aren't actually trying to decide whether to believe $H_0$ vs "$H_1$: the two groups have different means." Our test is merely a rudimentary threshold for checking if we have enough data to rank the two means, hoping to rule out sampling variation as a major concern. Rejecting or failing to reject $H_0$ doesn't directly tell us what to believe about treatment & control, nor what real-world policy to follow. Instead it tells us about the study design: whether the sample size was minimally adequate, or whether it was so small that this sample can't be trusted for the purpose of ranking the two groups' means.

Even if you do reject $H_0$, it really isn't safe to rank the two groups' means unless you've also ruled out the other concerns besides sampling variation (confounding, bias, measurement error, etc.)

---

Same thing with bigger models. You fit a regression model to predict $Y$ from a whole bunch of $X$ variables, and maybe your substantive question is whether $X_1$ is positively or negatively associated with $Y$ after controlling for the other $X_2, X_3$, and so on. If you test "$H_0: \beta_1=0$ in this particular model" vs "$H_1: \beta_1\neq 0$ in this particular model," you haven't carved up the whole world into two exclusive possibilities---it's quite likely that BOTH models are oversimplifications and therefore "false"---so rejecting one doesn't really imply believing the other. And regardless, the test isn't designed to tell you which model is more plausible or believable. All it can tell you is whether you seem to have enough data & can afford to fit the larger model, so that it's safe (purely in terms of sampling variation concerns) to interpret the sign of $\hat\beta_1$... or if you can't even afford to do that, because at your limited sample size the sampling variation has a high risk of misleading you & giving you a sample in which $\hat\beta_1$ has the wrong sign compared to the population's true $\beta_1$.