If all 1000 test patients are not cured by the drug, can't we say that we accept the null hypothesis?

Question

In many places I have read that we can never say that we "accept" the null hypothesis. Instead we must say that we "fail to reject" the null hypothesis.

But I don't see how that squares with this simple example: Suppose we are testing a drug that is supposed to cure diabetes completely within 24 hours. We try it on 1000 patients, and all of them still have diabetes after taking the drug.

Isn't it obvious that this drug doesn't cure diabetes? i.e., that we accept the null hypothesis?

I certainly wouldn't put my faith in this drug.

---

Null hypothesis: The drug has no effect on the patients.

Alternative hypothesis: The drug cures diabetes

Answer 1

Possibility one: The drug has a very small effect. Perhaps it cures .0001% of people taking it. The test you outlined only implies there is not enough evidence for the dramatic alternative you have proposed.

Possibility two: The drug has a very strong negative effect. (credit to @ssdecontrol) Perhaps the drug has no effect and all of those patients would have gotten better on their own, but due to the drug none of the patients recovered.

Without any prior knowledge, the data would be consistent with these possibilities as well as with the possibility that the null is true.

So, failing to reject the null does not imply that the null is any more true than these other possibilities.

Answer 2

There are some good answers here, but what I think is the key issue is not explicitly stated anywhere. In short, your formulation of the null and alternative hypotheses is invalid. The null and alternative hypotheses must be mutually exclusive (that is, they cannot both be true). Your formulation meets that criterion. However, they must also be collectively exhaustive (that is, one of them has to be true). Your formulation does not meet this criterion.

You cannot have a null hypothesis that the drug has a $0\%$ chance of curing diabetes and an alternative hypothesis that the drug has a $100\%$ chance of curing diabetes. Imagine that the true probability the drug will cure diabetes is $50\%$, then both your null and your alternative hypotheses are false. That is your problem.

The prototypical null hypothesis is a point value (e.g., $0$ on the real number line, or most often $50\%$ when referring to probabilities, but those are just conventions). In addition, if you are working with a bounded parameter space (as you are here—probabilities must range within $[0,\ 1]$), it is generally problematic to try to test values that are at the limits (i.e., $0$ or $1$). Having chosen a point value as your null (the value you want to reject), you can get evidence against it, but cannot get evidence for it from your data (cf. @John's insightful answer). To understand this further, it may help you to read my answer here: Why do statisticians say a non-significant result means “you can't reject the null” as opposed to accepting the null hypothesis? To apply those ideas to your situation more concretely, even if your null were $0\%$ (and hence your alternative hypothesis was $\pi\ne 0$), and you had tried the drug on $100,\!000$ patients without a single one being cured, you could not accept your null hypothesis: The data would still be consistent with the possibility that the probability was $0.00003$ (see: How to tell the probability of failure if there were no failures?).

On the other hand, you don't have to have a point null. One-tailed (i.e., $< \theta_0$) null hypotheses are not points, for instance. They are sets of infinite points. Likewise, you could also have a range / interval hypothesis (e.g., that the parameter is within $[a,\ b]$). In that case, you can accept your null on the basis of the evidence—that is what equivalence testing is all about. (You can still be making a type I error, of course.)

Answer 3

As the other users have commented, the issue with accepting the null hypothesis is that we don't have enough evidence (nor will we ever) to conclude that the effect is exactly 0. Mathematically, hypothesis testing is generally not capable of answering such questions.

However, that does not mean that the intent of your question is not a valid one! In fact, this is typically the intent in clinical trials for drug generics: the goal is not to show that you've produced a more effective drug, but rather that your drug is essentially about as effective as the name brand (and you can produce it at a much lower cost). Equivalency is typically thought of as the null hypothesis.

To address this question using hypothesis testing, the question is reformed in such as a way that it can be answered. The reformatted question looks something like this:

$H_o: \beta_g \leq \beta_{nb}\times 0.75$

$H_a: \beta_g > \beta_{nb} \times 0.75$

where $\beta_g$ is the effect of generic and $\beta_{nb}$ is the effect of the name brand drug. So now if we reject the null hypothesis, we can conclude the generic is at least 75% as effective as the namebrand. Clearly, this is not the same as saying exactly equivalent, but it gets at the question you're interested in (and in a way that I believe is a mathematically more reasonable question).

We can approach your question in a similar manner. Rather than try to say "do we have enough evidence to conclude 0 effect?", we can ask "given our evidence, what is the maximum effect for which our results were not too unusual?". With $n = 1000$ and 0 successes, can claim we have enough evidence to conclude that the probability of success is less than 0.3% (based on Fisher's exact test, $\alpha = 0.05$).

From this result, surely you can still conclude that this is not a drug you will have faith in.

Answer 4

Suppose that the drug works, but only on .00001% of the population. The drug works, period. What are the odds of detecting, statistically, that it works it a sample of 10000 people? 100,000 people? 1,000,000 people?

Answer 5

It is incorrect to say that you cannot ever accept the null hypothesis. You're taking the textbook information out of context. What you can't do is use a null hypothesis test to accept it. The test is for rejecting the hypothesis. Note that your own argument for accepting has little to do with a test outcome. It's about the data. It would be rather inane to run a test at all in your example. You can use your data to argue that you accept the null hypothesis. There's nothing wrong with that. You just can't use the results of the test to do so.

The reason that you can't use a hypothesis test all by itself is because it's not designed to do that. If you're not understanding that from the textbooks it's understandable. It's actually an interesting paradox that the p-value only actually means something if the null is true but can't be used to demonstrate the null is true. To make it easier perhaps just consider power sensitivity. You could always just collect far too few samples and fail to reject the null. Since you can do that it's clear the test alone isn't a valid reason to accept the null. But again, that doesn't mean you can never say the null is true. It only means the test is no foundation for arguing the null is true.

NOTE: There is an Occam's razor argument that you should accept the null when you don't reject; but the test isn't telling you to accept the null. What you're doing is accepting the null as the default and if you don't reject with the test then you're maintain the default state. So even in this case the null is not accepted because of the test.

Answer 6

Looking through your comments, I think that you are very interested in this question: why can we accumulate enough evidence to reject the null, but not the alternative, i.e. what makes hypothesis testing a ones-sided street?

The very important thing to think about is what values constitutes the null hypothesis? In your example, it is only a single value, $i.e.$, $p = 0$. The alternative, conversely, is $p > 0$.

We accept either hypothesis if all "reasonable values" (i.e. values inside our confidence interval) fall completely into the range given by that hypothesis. So if all our reasonable values are greater than 0, we would accept the alternative. On the other hand, the null hypothesis is only a single point, 0! So to accept the null, we would have to be have confidence interval of length 0. Since (generally speaking) the length confidence interval approaches 0 as $n \rightarrow \infty$, but doesn't achieve length 0 for finite $n$, we would need to collect an infinite amount of data to conclude that we have no margin of error in our estimate.

But note that if we define the null hypothesis to be more than just a single point, i.e. a one sided hypothesis test such as

$H_o: p \leq 0.5$

$H_a: p > 0.5$

we actually can accept the null hypothesis. Suppose our confidence interval were to be (0.35, 0.45). All these values less than or equal to 0.5, which is in the region of the null hypothesis. So in that case, we could accept the null.

Small, technical, abuse of statistics note: if one is really willing to abuse asymptotic theory, one actually could (but shouldn't...) accept the null in your example: the asymptotic standard error is $\sqrt{(\hat p (1 - \hat p) /n)} = 0$. So your asymptotic confidence interval will be (0,0), all of which belongs to the null hypothesis. But this just abusing asymptotic results; note that you get the same conclusion even if $n$ = 1.

Answer 7

I know you are dealing with null hypothesis, but the real problem is the example given or as stated the Simple Example. 1,000 people are given a drug and it doesn't work. What other maladies did these people have, what were their ages and stages of desease. To declare a null hypothesis more information; probably detailed ; must be given to make this work in a scientific setting.