9
$\begingroup$

In many places I have read that we can never say that we "accept" the null hypothesis. Instead we must say that we "fail to reject" the null hypothesis.

But I don't see how that squares with this simple example: Suppose we are testing a drug that is supposed to cure diabetes completely within 24 hours. We try it on 1000 patients, and all of them still have diabetes after taking the drug.

Isn't it obvious that this drug doesn't cure diabetes? i.e., that we accept the null hypothesis?

I certainly wouldn't put my faith in this drug.


Null hypothesis: The drug has no effect on the patients.

Alternative hypothesis: The drug cures diabetes

$\endgroup$
  • 6
    $\begingroup$ What exactly are the null & alternative hypotheses supposed to be in this scenario? Moreover, how is this scenario supposed to relate / generalize to other (realistic) situations & the logic of hypothesis testing? $\endgroup$ – gung - Reinstate Monica Jun 27 '15 at 22:09
  • $\begingroup$ @gung The null hypothesis is that the drug has no effect on the patients. The alternative hypothesis is that the drug cures diabetes. Isn't it obvious that the null hypothesis is true? $\endgroup$ – Jonathan Aquino Jun 28 '15 at 1:09
  • $\begingroup$ @gung This relates to the logic of hypothesis testing because I am trying to understand why we can never say that we "accept" the null hypothesis and yet in this case it seems that we can say that we "accept" the null hypothesis. $\endgroup$ – Jonathan Aquino Jun 28 '15 at 1:14
  • 4
    $\begingroup$ While in some cases the situation may be "obvious", if you want to use the language of hypothesis testing to justify a claim, you should also keep to its reasoning. If something is obvious, describe that obvious thing (e.g. "It's obvious there's no practical benefit from the drug"; avoiding statistical terms avoids the need for statistical arguments) $\endgroup$ – Glen_b Jun 28 '15 at 1:31
  • 2
    $\begingroup$ This is a case where a confidence interval or estimate of effects size may be more informative. Reality is not binary, the drug may work in a philosophical sense (i.e., it does increase cure rate), but be so meager as to be practically negligible. Hypothesis testing is too crude of a tool for this, but CIs and effects sizes can get you there. If the CI is very narrow about 0, then any possible benefit is likely to be very small. $\endgroup$ – user75138 Jun 28 '15 at 5:06
15
$\begingroup$

Possibility one: The drug has a very small effect. Perhaps it cures .0001% of people taking it. The test you outlined only implies there is not enough evidence for the dramatic alternative you have proposed.

Possibility two: The drug has a very strong negative effect. (credit to @ssdecontrol) Perhaps the drug has no effect and all of those patients would have gotten better on their own, but due to the drug none of the patients recovered.

Without any prior knowledge, the data would be consistent with these possibilities as well as with the possibility that the null is true.

So, failing to reject the null does not imply that the null is any more true than these other possibilities.

$\endgroup$
  • 4
    $\begingroup$ Alternatively, what if the drug has no effect and all of those patients would have gotten better on their own, anyway. $\endgroup$ – shadowtalker Jun 28 '15 at 2:10
  • 1
    $\begingroup$ I think this answer is missing a piece. You're right that these possibilities could be true, but it's also true that no one can ever really know anything. This is why we're willing to reject the null once it reaches a certain degree of improbability, rather than waiting for unattainable, definitive proof. But if that's true, why aren't we willing to accept the null once we've accumulated a certain degree of evidence? $\endgroup$ – octern Jun 28 '15 at 12:37
  • $\begingroup$ Jason - I'd be interested in hearing your response to @octern's comment above. $\endgroup$ – Jonathan Aquino Jun 28 '15 at 18:08
  • $\begingroup$ @octern Very good question. What if the null hypothesis was that the drug had a a very, very, very small positive effect? We would also fail to reject the null. Obviously we shouldn't publish a paper saying the drug has a very, very, very small positive effect. But then it would be equally suspect to publish a paper saying that the drug has zero effect. The data is consistent with many possibilities and we don't know which of the remaining possibilities is true. See (the formal logical fallacy)[en.wikipedia.org/wiki/Argument_from_ignorance] $\endgroup$ – Jason Sanchez Jun 29 '15 at 0:41
  • $\begingroup$ I see, you're quite right. I now understand a little more about why frequentist hypothesis testing always deals with rejecting hypotheses rather than accepting them. So, the NHST approach to accepting a hypothesis of zero effect would be to 1) determine how small an effect can count as zero, and 2) gather enough evidence to reject the hypothesis that there is an effect of at least that size (really two hypotheses, one in each direction). Yes? $\endgroup$ – octern Jun 29 '15 at 4:51
6
$\begingroup$

There are some good answers here, but what I think is the key issue is not explicitly stated anywhere. In short, your formulation of the null and alternative hypotheses is invalid. The null and alternative hypotheses must be mutually exclusive (that is, they cannot both be true). Your formulation meets that criterion. However, they must also be collectively exhaustive (that is, one of them has to be true). Your formulation does not meet this criterion.

You cannot have a null hypothesis that the drug has a $0\%$ chance of curing diabetes and an alternative hypothesis that the drug has a $100\%$ chance of curing diabetes. Imagine that the true probability the drug will cure diabetes is $50\%$, then both your null and your alternative hypotheses are false. That is your problem.

The prototypical null hypothesis is a point value (e.g., $0$ on the real number line, or most often $50\%$ when referring to probabilities, but those are just conventions). In addition, if you are working with a bounded parameter space (as you are here—probabilities must range within $[0,\ 1]$), it is generally problematic to try to test values that are at the limits (i.e., $0$ or $1$). Having chosen a point value as your null (the value you want to reject), you can get evidence against it, but cannot get evidence for it from your data (cf. @John's insightful answer). To understand this further, it may help you to read my answer here: Why do statisticians say a non-significant result means “you can't reject the null” as opposed to accepting the null hypothesis? To apply those ideas to your situation more concretely, even if your null were $0\%$ (and hence your alternative hypothesis was $\pi\ne 0$), and you had tried the drug on $100,\!000$ patients without a single one being cured, you could not accept your null hypothesis: The data would still be consistent with the possibility that the probability was $0.00003$ (see: How to tell the probability of failure if there were no failures?).

On the other hand, you don't have to have a point null. One-tailed (i.e., $< \theta_0$) null hypotheses are not points, for instance. They are sets of infinite points. Likewise, you could also have a range / interval hypothesis (e.g., that the parameter is within $[a,\ b]$). In that case, you can accept your null on the basis of the evidence—that is what equivalence testing is all about. (You can still be making a type I error, of course.)

$\endgroup$
  • $\begingroup$ So a null hypothesis significance test (Ho: x = 0; Ha: x > 0) cannot support the null, but a one-tailed test (Ho: x <= δ; Ha: x > δ) can support the null. Do I have that correct? $\endgroup$ – Jonathan Aquino Jun 30 '15 at 4:08
  • $\begingroup$ @JonathanAquino, so long as $\theta_0\ne 0$ (or otherwise on the boundary of the parameter space) & thus ${\rm H}_0\!: x\le\theta$ is an infinite set of points, you could accept the null if the entire confidence interval was within the null interval. You should read the answers I linked to. $\endgroup$ – gung - Reinstate Monica Jun 30 '15 at 13:20
  • $\begingroup$ +1 long time ago but upon re-reading now I have a quibble: I don't think that H0 and H1 should necessarily be "collectively exhaustive", at least not in the Neyman-Pearson approach (where H1 usually corresponds to a particular effect size and this is used to do the power calculations). $\endgroup$ – amoeba says Reinstate Monica Dec 9 '16 at 23:28
  • $\begingroup$ @amoeba, for the sake of study planning & power analyses, you do have a specific effect size in mind, but the nature of hypothesis testing is H0: mu1 = m2; Ha: mu1 != mu2. That is the logical scheme of the hypothesis test. There are 2 different meanings of alternative hypothesis depending on the context. $\endgroup$ – gung - Reinstate Monica Dec 9 '16 at 23:39
2
$\begingroup$

As the other users have commented, the issue with accepting the null hypothesis is that we don't have enough evidence (nor will we ever) to conclude that the effect is exactly 0. Mathematically, hypothesis testing is generally not capable of answering such questions.

However, that does not mean that the intent of your question is not a valid one! In fact, this is typically the intent in clinical trials for drug generics: the goal is not to show that you've produced a more effective drug, but rather that your drug is essentially about as effective as the name brand (and you can produce it at a much lower cost). Equivalency is typically thought of as the null hypothesis.

To address this question using hypothesis testing, the question is reformed in such as a way that it can be answered. The reformatted question looks something like this:

$H_o: \beta_g \leq \beta_{nb}\times 0.75$

$H_a: \beta_g > \beta_{nb} \times 0.75$

where $\beta_g$ is the effect of generic and $\beta_{nb}$ is the effect of the name brand drug. So now if we reject the null hypothesis, we can conclude the generic is at least 75% as effective as the namebrand. Clearly, this is not the same as saying exactly equivalent, but it gets at the question you're interested in (and in a way that I believe is a mathematically more reasonable question).

We can approach your question in a similar manner. Rather than try to say "do we have enough evidence to conclude 0 effect?", we can ask "given our evidence, what is the maximum effect for which our results were not too unusual?". With $n = 1000$ and 0 successes, can claim we have enough evidence to conclude that the probability of success is less than 0.3% (based on Fisher's exact test, $\alpha = 0.05$).

From this result, surely you can still conclude that this is not a drug you will have faith in.

$\endgroup$
  • 2
    $\begingroup$ Well, if the success rate really were 0.3% and the drug was a glass of water or a ten-minute walk in the park, it could still be worth a general "prescription" $\endgroup$ – Hagen von Eitzen Jun 28 '15 at 20:09
1
$\begingroup$

Suppose that the drug works, but only on .00001% of the population. The drug works, period. What are the odds of detecting, statistically, that it works it a sample of 10000 people? 100,000 people? 1,000,000 people?

$\endgroup$
  • $\begingroup$ What would you say in response to @octern's comment? $\endgroup$ – Jonathan Aquino Jun 28 '15 at 18:15
1
$\begingroup$

It is incorrect to say that you cannot ever accept the null hypothesis. You're taking the textbook information out of context. What you can't do is use a null hypothesis test to accept it. The test is for rejecting the hypothesis. Note that your own argument for accepting has little to do with a test outcome. It's about the data. It would be rather inane to run a test at all in your example. You can use your data to argue that you accept the null hypothesis. There's nothing wrong with that. You just can't use the results of the test to do so.

The reason that you can't use a hypothesis test all by itself is because it's not designed to do that. If you're not understanding that from the textbooks it's understandable. It's actually an interesting paradox that the p-value only actually means something if the null is true but can't be used to demonstrate the null is true. To make it easier perhaps just consider power sensitivity. You could always just collect far too few samples and fail to reject the null. Since you can do that it's clear the test alone isn't a valid reason to accept the null. But again, that doesn't mean you can never say the null is true. It only means the test is no foundation for arguing the null is true.

NOTE: There is an Occam's razor argument that you should accept the null when you don't reject; but the test isn't telling you to accept the null. What you're doing is accepting the null as the default and if you don't reject with the test then you're maintain the default state. So even in this case the null is not accepted because of the test.

$\endgroup$
1
$\begingroup$

Looking through your comments, I think that you are very interested in this question: why can we accumulate enough evidence to reject the null, but not the alternative, i.e. what makes hypothesis testing a ones-sided street?

The very important thing to think about is what values constitutes the null hypothesis? In your example, it is only a single value, $i.e.$, $p = 0$. The alternative, conversely, is $p > 0$.

We accept either hypothesis if all "reasonable values" (i.e. values inside our confidence interval) fall completely into the range given by that hypothesis. So if all our reasonable values are greater than 0, we would accept the alternative. On the other hand, the null hypothesis is only a single point, 0! So to accept the null, we would have to be have confidence interval of length 0. Since (generally speaking) the length confidence interval approaches 0 as $n \rightarrow \infty$, but doesn't achieve length 0 for finite $n$, we would need to collect an infinite amount of data to conclude that we have no margin of error in our estimate.

But note that if we define the null hypothesis to be more than just a single point, i.e. a one sided hypothesis test such as

$H_o: p \leq 0.5$

$H_a: p > 0.5$

we actually can accept the null hypothesis. Suppose our confidence interval were to be (0.35, 0.45). All these values less than or equal to 0.5, which is in the region of the null hypothesis. So in that case, we could accept the null.

Small, technical, abuse of statistics note: if one is really willing to abuse asymptotic theory, one actually could (but shouldn't...) accept the null in your example: the asymptotic standard error is $\sqrt{(\hat p (1 - \hat p) /n)} = 0$. So your asymptotic confidence interval will be (0,0), all of which belongs to the null hypothesis. But this just abusing asymptotic results; note that you get the same conclusion even if $n$ = 1.

$\endgroup$
0
$\begingroup$

I know you are dealing with null hypothesis, but the real problem is the example given or as stated the Simple Example. 1,000 people are given a drug and it doesn't work. What other maladies did these people have, what were their ages and stages of desease. To declare a null hypothesis more information; probably detailed ; must be given to make this work in a scientific setting.

$\endgroup$
  • $\begingroup$ No matter how much information we add - whether ages or stages of disease - we can never accept the null hypothesis. I am trying to understand why. $\endgroup$ – Jonathan Aquino Jun 28 '15 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.