Is it possible to prove a null hypothesis?

Question

As the question states - Is it possible to prove the null hypothesis? From my (limited) understanding of hypothesis, the answer is no but I can't come up with a rigorous explanation for it. Does the question have a definitive answer?

Answer 1

If you are talking about the real world & not formal logic, the answer is of course. "Proof" of anything by empirical means depends on the strength of the inference one can make, which in turn is determined by validity of the testing process as evaluated in light of everything one knows about how the world works (i.e., theory). Whenever one accepts that certain empirical results justify rejecting the "null" hypothesis, one is necessarily making judgments of this sort (validity of design; world works in certain way), so having to make the analogous assumptions necessary to justify inferring "proof of the null" is not problematic at all.

So what are the analogous assumptions? Here is an example of "proving the null" that is commonplace in health science & in social science. (1) Define "null" or "no effect" in some way that is practically meaningful. Let's say that I believe that I should conduct myself as if there is no meaningful difference between 2 treatments, t1 & t2, for a disease unless one gives a 3% better chance of recovery than the other. (2) Figure out a valid design for testing whether there is any effect-- in this case, whether there is a difference in recovery likelihood between t1 & t2. (3) Do a power analysis to determine whether what sample size is necessary to generate a sufficiently high likelihood-- one that I am confident relying on given what's at stake -- that I would see the meaningful effect, 3% in my example, assuming it exists. Usually people say power is sufficient if the likelihood of observing a specified effect at a specified alpha is at least 0.80, but the right level of confidence is really a matter of how averse you are to error -- same as it is when you select p-value threshold for "rejecting the null."(4) Perform the empirical test & observe the effect. If it is below the specified "meaningful difference" value -- 3% in my example -- you've "proven" that there is "no effect."

For a good treatment of this matter, see Streiner, D.L. Unicorns Do Exist: A Tutorial on “Proving” the Null Hypothesis. Canadian Journal of Psychiatry 48, 756-761 (2003).

Answer 2

Answer from the mathematical side : it is possible if and only if "hypotheses are mutually singular".

If by "prove" you mean have a rule that can "accept" (should I say that:) ) $H_0$ with a probability to make a mistake that is zero, then you are searching what could be called "ideal test" and this exists:

If you are testing wether a random variable $X$ is drawn from $P_0$ or from $P_1$ (i.e testing $H_0: X\leadsto P_0$ versus $H_1: X\leadsto P_1$) then there exists an ideal test if and only if $P_1\bot P_0$ ($P_1$ and $P_0$ are "mutually singular").

If you don't know what "mutually singular" means I can give you an example: $\mathcal{U}[0,1]$ and $\mathcal{U}[3,4]$ (uniforms on $[0,1]$ and $[3,4]$) are mutually singular. This means if you want to test

$H_0: X\leadsto \mathcal{U}[0,1]$ versus $H_1: X\leadsto \mathcal{U}[3,4]$

then there exist an ideal test (guess what it is :) ) : a test that is never wrong !

If $P_1$ and $P_0$ are not mutually singular, then this does not exist (this results from the "only if part")!

In non mathematical terms this means that you can prove the null if and only if the proof is already in your assumptions (i.e. if and only if you have chosen the hypothesis $H_0$ and $H_1$ that are so different that a single observation from $H_0$ cannot be identifyed as one from $H_1$ and vise versa).

Answer 3

Yes there is a definitive answer. That answer is: No, there isn't a way to prove a null hypothesis. The best you can do, as far as I know, is throw confidence intervals around your estimate and demonstrate that the effect is so small that it might as well be essentially non-existent.

Answer 4

For me, the decision theoretical framework presents the easiest way to understand the "null hypothesis". It basically says that there must be at least two alternatives: the Null hypothesis, and at least one alternative. Then the "decision problem" is to accept one of the alternatives, and reject the others (although we need to be precise about what we mean by "accepting" and "rejecting" the hypothesis). I see the question of "can we prove the null hypothesis?" as analogous to "can we always make the correct decision?". From a decision theory perspective the answer is clearly yes if

1)there is no uncertainty in the decision making process, for then it is a mathematical exercise to work out what the correct decision is.

2)we accept all the other premises/assumptions of the problem. The most critical one (I think) is that the hypothesis we are deciding between are exhaustive, and one (and only one) of them must be true, and the others must be false.

From a more philosophical standpoint, it is not possible to "prove" anything, in the sense that the "proof" depends entirely on the assumptions / axioms which lead to that "proof". I see proof as a kind of logical equivalence rather than a "fact" or "truth" in the sense that if the proof is wrong, the assumptions which led to it are also wrong.

Applying this to the "proving the null hypothesis" I can "prove" it to be true by simply assuming that it is true, or by assuming that it is true if certain conditions are meet (such as the value of a statistic).

Answer 5

Technically, no, a null hypothesis cannot be proven. For any fixed, finite sample size, there will always be some small but nonzero effect size for which your statistical test has virtually no power. More practically, though, you can prove that you're within some small epsilon of the null hypothesis, such that deviations less than this epsilon are not practically significant.

Answer 6

There is a case where a proof is possible. Suppose you have a school and your null hypothesis is that the numbers of boys and of girls is equal. As the sample size increases, the uncertainty in the ratio of boys to girls tends to reduce, eventually reaching certainty (which is what I assume you mean by proof) when the whole pupil population is sampled.

But if you do not have a finite population, or if you are sampling with replacement and cannot spot resampled individuals, then you cannot reduce the uncertainty to zero with a finite sample.

Answer 7

Yes, it is possible to prove the null--in exactly the same sense that it is possible to prove any alternative to the null. In a Bayesian analysis, it is perfectly possible for the odds in favor of the null versus any of the proposed alternatives to it to become arbitrarily large. Moreover, it is false to assert, as some of the above answers assert, that one can only prove the null if the alternatives to it are disjoint (do not overlap with the null). In a Bayesian analysis every hypothesis has a prior probability distribution. This distribution spreads a unit mass of prior probability out over the proposed alternatives. The null hypothesis puts all of the prior probability on a single alternative. In principle, alternatives to the null may put all of the prior probability on some non-null alternative (on another "point"), but this is rare. In general, alternatives hedge, that is, they spread the same mass of prior probability out over other alternatives--either to the exclusion of the null alternative, or, more commonly, including the null alternative. The question then becomes which hypothesis puts the most prior probability where the experimental data actually fall. If the data fall tightly around where the null says they should fall, then it will be the odds-on favority (among the proposed hypotheses) EVEN THOUGH IT IS INCLUDED IN (NESTED IN, NOT MUTUALLY EXCLUSIVE WITH) THE ALTERNATIVES TO IT. The believe that it is not possible for a nested alternative to be more likely than the set in which it is nested reflects the failure to distinguish between probability and likelihood. While it is impossible for a component of a set to be less probable than the entire set, it is perfectly possible for the posterior likelihood of a component of a set of hypotheses to be greater than the posterior likelihood of the set as a whole. The posterior likelihood of an hypothesis is the product of the likelihood function and the prior probability distribution that the hypothesis posits. If an hypothesis puts all of the prior probability in the right place (e.g., on the null), then it will have a higher posterior likelihood than an hypothesis that puts some of the prior probability in the wrong place (not on the null).