# Is it possible to prove a null hypothesis?

As the question states - Is it possible to prove the null hypothesis? From my (limited) understanding of hypothesis, the answer is no but I can't come up with a rigorous explanation for it. Does the question have a definitive answer?

-
It depends on what you mean by "prove." As stated, this is a philosophical question, not a statistical one, and has no definitive answer (although, at least since David Hume's time, most people would answer "no"). –  whuber Jan 13 '11 at 16:54
This is somewhat of an ill-posed question. We need to know the conditions under which this "proof" is to occur. –  probabilityislogic Jan 16 '11 at 15:39
Perhaps a better posed question is "Under what conditions/assumptions is it possible to prove the null hypothesis?" –  probabilityislogic Jan 16 '11 at 15:40

If you are talking about the real world & not formal logic, the answer is of course. "Proof" of anything by empirical means depends on the strength of the inference one can make, which in turn is determined by validity of the testing process as evaluated in light of everything one knows about how the world works (i.e., theory). Whenever one accepts that certain empirical results justify rejecting the "null" hypothesis, one is necessarily making judgments of this sort (validity of design; world works in certain way), so having to make the analogous assumptions necessary to justify inferring "proof of the null" is not problematic at all.

So what are the analogous assumptions? Here is an example of "proving the null" that is commonplace in health science & in social science. (1) Define "null" or "no effect" in some way that is practically meaningful. Let's say that I believe that I should conduct myself as if there is no meaningful difference between 2 treatments, t1 & t2, for a disease unless one gives a 3% better chance of recovery than the other. (2) Figure out a valid design for testing whether there is any effect-- in this case, whether there is a difference in recovery likelihood between t1 & t2. (3) Do a power analysis to determine whether what sample size is necessary to generate a sufficiently high likelihood-- one that I am confident relying on given what's at stake -- that I would see the meaningful effect, 3% in my example, assuming it exists. Usually people say power is sufficient if the likelihood of observing a specified effect at a specified alpha is at least 0.80, but the right level of confidence is really a matter of how averse you are to error -- same as it is when you select p-value threshold for "rejecting the null."(4) Perform the empirical test & observe the effect. If it is below the specified "meaningful difference" value -- 3% in my example -- you've "proven" that there is "no effect."

For a good treatment of this matter, see Streiner, D.L. Unicorns Do Exist: A Tutorial on “Proving” the Null Hypothesis. Canadian Journal of Psychiatry 48, 756-761 (2003).

-
+1. This is a nice example of the importance of being clear about one's standard of "proof." In many applications the one you invoke here--the "act as if" standard, if I may call it that--is so weak that nobody would accept it as "proof." I do not deny its utility, though, and advocate this kind of approach to support rational decision making. (But maybe Bayesian methods are better... :-) –  whuber Jan 14 '11 at 0:47
(+1) Nice answer. I added a link to an online version of Streiner's article; I hope you don't mind (feel free to remove). –  chl Jan 14 '11 at 9:18
couple more things: (1) Treating failure to reject the null as evidence in support of null is a shockingly common error & usual occasion for Streiner's point. This mistake essentially turns the strong aversion to type 1 error in "p < 0.05" norm into license to make type 2. S says, "wait--you need power..." (2) Whuber cites Hume's famous argument. H's pt is actually just as subversive of empirical proofs rejecting the null as of proofs of the null. H says induction can't support causal inference. Ok; but there's no alternative for empirical study! Go Pearl (& Bayes), not Hume, on causality! –  dmk38 Jan 14 '11 at 23:49
this question on equivalence testing also has some good suggestions stats.stackexchange.com/questions/3038/… –  Jeromy Anglim Feb 11 '11 at 4:46

Answer from the mathematical side : it is possible if and only if "hypotheses are mutually singular".

If by "prove" you mean have a rule that can "accept" (should I say that:) ) $H_0$ with a probability to make a mistake that is zero, then you are searching what could be called "ideal test" and this exists:

If you are testing wether a random variable $X$ is drawn from $P_0$ or from $P_1$ (i.e testing $H_0: X\leadsto P_0$ versus $H_1: X\leadsto P_1$) then there exists an ideal test if and only if $P_1\bot P_0$ ($P_1$ and $P_0$ are "mutually singular").

If you don't know what "mutually singular" means I can give you an example: $\mathcal{U}[0,1]$ and $\mathcal{U}[3,4]$ (uniforms on $[0,1]$ and $[3,4]$) are mutually singular. This means if you want to test

$H_0: X\leadsto \mathcal{U}[0,1]$ versus $H_1: X\leadsto \mathcal{U}[3,4]$

then there exist an ideal test (guess what it is :) ) : a test that is never wrong !

If $P_1$ and $P_0$ are not mutually singular, then this does not exist (this results from the "only if part")!

In non mathematical terms this means that you can prove the null if and only if the proof is already in your assumptions (i.e. if and only if you have chosen the hypothesis $H_0$ and $H_1$ that are so different that a single observation from $H_0$ cannot be identifyed as one from $H_1$ and vise versa).

-
+1 Nice answer. A simple rendering of the math is that the null and its alternatives are assumed to yield disjoint sets of outcomes; e.g., either there is a zebra in this room or there isn't. Of course "prove" here implicitly includes "conditional on the model," which itself is never established with the same rigor as, say, a mathematical theorem; it implicitly includes "conditional on the accuracy of the observations;" and it implicitly includes that the hypotheses can be unambiguously interpreted. (For criticism of the latter, see George Lakoff's Women, Fire, and Dangerous Things.) –  whuber Jan 13 '11 at 21:29

Yes there is a definitive answer. That answer is: No, there isn't a way to prove a null hypothesis. The best you can do, as far as I know, is throw confidence intervals around your estimate and demonstrate that the effect is so small that it might as well be essentially non-existent.

-
More generally the problem in statistics is not that you are unable to prove the null hypothesis, it is that you are not able to make any point estimates with certainty. That is, just as you can't say "there is no effect of the variable" you are unable to say that "the effect size of the variable is 1.95". Statistics always have confidence intervals. –  rpierce Jan 20 '11 at 3:41

For me, the decision theoretical framework presents the easiest way to understand the "null hypothesis". It basically says that there must be at least two alternatives: the Null hypothesis, and at least one alternative. Then the "decision problem" is to accept one of the alternatives, and reject the others (although we need to be precise about what we mean by "accepting" and "rejecting" the hypothesis). I see the question of "can we prove the null hypothesis?" as analogous to "can we always make the correct decision?". From a decision theory perspective the answer is clearly yes if

1)there is no uncertainty in the decision making process, for then it is a mathematical exercise to work out what the correct decision is.

2)we accept all the other premises/assumptions of the problem. The most critical one (I think) is that the hypothesis we are deciding between are exhaustive, and one (and only one) of them must be true, and the others must be false.

From a more philosophical standpoint, it is not possible to "prove" anything, in the sense that the "proof" depends entirely on the assumptions / axioms which lead to that "proof". I see proof as a kind of logical equivalence rather than a "fact" or "truth" in the sense that if the proof is wrong, the assumptions which led to it are also wrong.

Applying this to the "proving the null hypothesis" I can "prove" it to be true by simply assuming that it is true, or by assuming that it is true if certain conditions are meet (such as the value of a statistic).

-

Technically, no, a null hypothesis cannot be proven. For any fixed, finite sample size, there will always be some small but nonzero effect size for which your statistical test has virtually no power. More practically, though, you can prove that you're within some small epsilon of the null hypothesis, such that deviations less than this epsilon are not practically significant.

-