# Which one is the null hypothesis? Conflict between science theory, logic and statistics?

I am having difficulties understanding the underlying logic in setting the null hypothesis. In this answer the obviously generally accepted proposition is stated that the null hypothesis is the hypothesis that there will be no effect, everything stays the same, i.e. nothing new under the sun, so to speak.

The alternative hypothesis is then what you try to prove, that e.g. a new drug delivers on its promises.

Now coming form science theory and general logic we know that we can only falsify propositions, we cannot prove something (no number of white swans can prove that all swans are white but one black swan can disprove it). This is why we try to disprove the null hypothesis, which is not equivalent to proving the alternative hypothesis - and this is where my skepticism starts - I will give an easy example:

Let's say I want to find out what kind of animal is behind a curtain. Unfortunately I cannot directly observe the animal but I have a test which gives me the number of legs of this animal. Now I have the following logical reasoning:

If the animal is a dog then it will have 4 legs.

If I conduct the test and find out that it has 4 legs this is no proof that it is a dog (it can be a horse, a rhino or any other 4-legged animal). But if I find out that it has not 4 legs this is a definite proof that it can not be a dog (assuming a healthy animal).

Translated into drug effectiveness I want to find out if the drug behind the curtain is effective. The only thing I will get is a number that gives me the effect. If the effect is positive, nothing is proved (4 legs). If there is no effect, I disprove the effectiveness of the drug.

Saying all this I think - contrary to common wisdom - the only valid null hypothesis must be

The drug is effective (i.e.: if the drug is effective you will see an effect).

because this is the only thing that I can disprove - up to the next round where I try to be more specific and so on. So it is the null hypothesis that states the effect and the alternative hypothesis is the default (no effect).

Why is it that statistical tests seem to have it backwards?

P.S.: You cannot even negate the above hypothesis to get a valid equivalent hypothesis, so you cannot say "The drug is not effective" as a null hypothesis because the only logically equivalent form would be "if you see no effect the drug will not be effective" which brings you nowhere because now the conclusion is what you want to find out!

P.P.S.: Just for clarification after reading the answers so far: If you accept scientific theory, that you can only falsify statements but not prove them, the only thing that is logically consistent is choosing the null hypothesis as the new theory - which can then be falsified. Because if you falsify the status quo you are left empty handed (the status quo is disproved but the new theory far from being proved!). And if you fail to falsify it you are in no better position either.

-
Hint: "The drug is effective" has not been sufficiently quantified to be either a scientific or a statistical statement. How are you going to make it quantitative? –  whuber Aug 3 '11 at 14:20
@whuber: This is the least of my problems: Just say that e.g. blood preasure is lowered by 10%. I argue that this must be the null hypothesis - the alternative hypothesis is "Nothing happens". –  vonjd Aug 3 '11 at 14:25
On the contrary, this is the crux of the question. It's perfectly fine in statistics to posit a null that says the effect is -10%. Your experiment will be able to reject it if it produces strong enough evidence to the contrary. Note, though, that (barring extraordinary computational and conceptual machinations) you can test only a single such hypothesis per experiment. Note, too, that it's the rare experimenter who knows so precisely what the effect size will be (but still feels a need to test it!). –  whuber Aug 3 '11 at 14:30
Well, in practice with drug trials the null is usually construed as "the drug is no more effective than the current treatment" and the alternative is "the drug is more effective than the current treatment." That has a built-in effect size, incidentally. With this formulation, evidence for the efficacy of the drug can reject the null. Upon swapping the hypotheses, evidence for the efficacy merely discourages one from rejecting the claim that the drug is good. In the first case the burden of proof is far more stringent. –  whuber Aug 3 '11 at 15:11
@vonjd: You say, "if you falsify the status quo you are left empty handed". Wrong. If we were making qualitative judgments "dog" / "not dog" it is true that providing evidence "not dog" is not particularly strong evidence for "dog". However, this is the value of quantifying things. If I provide evidence of "not 0" it provides good evidence for the value being something other than 0. If you are concerned that provides equal evidence for a good effect and bad effect, use a one-tailed test. –  rpierce Aug 3 '11 at 23:16

In statistics there are tests of equivalence as well as the more common test the Null and decide if sufficient evidence against it. The equivalence test turn this on its head and posits that effects are different as the Null and we determine if their is sufficient evidence against this Null.

I'm not clear on your drug example. If the response is a value/indicator of the effect, then an effect of 0 would indicate not effective. One would set that as the Null and evaluate the evidence against this. If the effect is sufficiently different from zero we would conclude that the no-effectiveness hypothesis is inconsistent with the data. A two-tailed test would count sufficiently negative values of effect as evidence against the Null. A one tailed test, the effect is positive and sufficiently different from zero, might be a more interesting test.

If you want to test if the effect is 0, then we'd need to flip this around and use an equivalence test where the H0 is the effect is not equal to zero, and the alternative is that H1 = the effect = 0. That would evaluate the evidence against the idea that effect was different from 0.

-
Part of the issue here is that, IIRC, the reason we select the no-effect as the Null is because the parameter for that effect is known, it is 0. If you want to turn this around and have some non-zero effect as the Null, then we'd need to know in advance what the value of this parameter was for the entire population, and if we knew the value of the parameter for the population there would be no point in testing. –  Gavin Simpson Aug 3 '11 at 11:37
Well, it seems that we will be having the same problem with the alternative hypothesis (we don't know the parameter there either). So my question is: Why not swapping both? This seems logically more consistent. –  vonjd Aug 3 '11 at 11:57
I'll let others comment on equivalence tests. They aren't the same as just swapping the hypotheses in standard tests, but I'm not that familiar with those ideas. I don't think you are correct that the equivalence tests suffer from the problem I mention in the comments. They are formulated from a very different theoretical view point. –  Gavin Simpson Aug 3 '11 at 12:06

First, I've heard this idea that propositions can only be falsified, but never proven. Could you post a link to a discussion of this, because with our wording here it doesn't seem to hold up very well - if X is a proposition, then not(X) is a proposition too. If disproving propositions is possible, then disproving X is the same as proving not(X), and we've proven a proposition.

Second, your analogy between the P(effective|$test_+$) and P(dog|4 legs) is interesting. The wording should be changed a little bit though:

The drug is effective (i.e.: iff the drug is effective you will see an effect).

In fact, P(effective|$test_+$) is often greater than P($test_+$|effective), as long as you use hypothesis testing and the right statistical model. Hypothesis testing formalizes the unlikelihood of positive test results under $H_0$. But an effective drug doesn't guarentee a positive test; when the drug is effective and variance is high the effect can be masked in the test.

If you observe $test_+$ you can infer effectiveness, because the alternative is $H_0$, and the hypothesis testing is set up so that P($test_+$|$H_0$) < 0.05.

So the difference between the dog case and the effectiveness case is in the appropriateness of the inference from the evidence to the conclusion. In the dog case, you have observed some evidence that doesn't strongly imply a dog. But in the clinical trial case you have observed some evidence that does strongly imply efficacy.

-
Thank you. If you accept that you can only falsify statements but not prove them (link in a second) the only thing that is logically consistent is choosing the null hypothesis as the new theory - which can then be falsified. If you falsify the status quo you are left empty handed (the status quo is disproved but the new theory far from being proved!). Now for the link, I think a good starting point would be: en.wikipedia.org/wiki/Falsifiability –  vonjd Aug 3 '11 at 14:05
I think a point to mention here is that you are not proving or disproving the null hypothesis. The decision you are making (classically) are to retain or to reject the null hypothesis. When you reject the null hypothesis you are not disproving it. All you are doing is saying that, given the observed data, the null hypothesis is unlikely. –  rpierce Aug 3 '11 at 14:39
@drknexus: Well, wouldn't you agree that this is the probabilistic equivalent of falsification in logic? –  vonjd Aug 3 '11 at 14:45
@drknexus Wouldn't it be more accurate to not say "given the observed data, the null hypothesis is unlikely" but rather "if the null hypothesis is true then this data is unlikely"? Isn't conflating those two the classic mistake in statistical hypothesis testing? –  Michael McGowan Aug 3 '11 at 18:03
MM: You're correct. I got sloppy in my wording. –  rpierce Aug 3 '11 at 23:09

You are right that, in a sense, frequentist hypothesis testing has it backwards. I'm not saying that that approach is wrong, but rather that the results are often not designed to answer the questions that the researcher is most interested in. If you want a technique more similar to the scientific method, try Bayesian inference.

Instead of talking about a "null hypothesis" that you can reject or fail to reject, with Bayesian inference you begin with a prior probability distribution based upon your understanding of the situation at hand. When you acquire new evidence, Bayesian inference provides a framework for you to update your belief with the evidence taken into account. I think this is how more similar to how science works.

-

I think you've got a fundamental error here (not that the whole area of hypothesis testing is clear!) but you say the alternative is what we try to prove. But this is not right. We attempt to reject (falsify) the null. If the results we obtain would be very unlikely if the null were true, we reject the null.

Now, as others said, this is not usually the question we want to ask: We don't usually care how likely the results are if the null is true, we care how likely the null is, given the results.

-

I think this is another case where frequentist statistics can't give a direct answer to the question you actually want to ask, and so answers a (no so) subtly different question, and it is easy to misinterpret this as a direct answer to the question you actually wanted to ask.

What we would really like to ask is normally what is the probability that the alternative hypothesis is true (or perhaps how much more likely to be true is it than the null hypothesis). However a frequentist analysis fundamentally cannot answer this question, as to a frequentist a probability is a long run frequency, and in this case we are interested in the truth of a particular hypothesis, which doesn't have a long run frequency - it is either true or it isn't. A Bayesian on the other hand can answer this question directly, as to a a Bayesian a probability is a measure of the plausibility of some proposition, so it is perfectly reasonable in a Bayesian analysis to assign a probability to the truth of a particular hypothesis.

The way frequentists deal will particular events is to treat them as a sample from some (possibly fictitious) population and make a statement about that population in place of a statement about the particular sample. For example, if you want to know the probability that a particular coin is biased, after observing N flips and observing h heads and t tails, a frequentist analysis cannot answer that question, however they could tell you the proportion of coins from a distribution of unbiased coins that would give h or more heads when flipped N times. As the natural definition of a probability that we use in everyday life is generally a Bayesian one, rather than a frequentist one, it is all too easy to treat this as the pobability that the null hypothesis (the coin is unbiased) is true.

Essentially frequentist hypothesis tests have an implicit subjectivist Bayesian component lurking at its heart. The frequentist test can tell you the likelihood of observing a statistic at least as extreme under the null hypothesis, however the decision to reject the null hypothesis on those grounds is entirely subjective, there is no rational requirement for you to do so. Essentiall experience has shown that we are generally on reasonably solid ground to reject the null if the p-value is suffciently small (again the threshold is subjective), so that is the tradition. AFAICS it doesn't fit well into the philosophy or theory of science, it is essentially a heuristic.

That doesn't mean it is a bad thing though, despite its imperfections frequentist hypothesis testing provides a hurdle that our research must get over, which helps us as scientists to keep our self-skepticism and not get carried away with enthusiasm for our theories. So while I am a Bayesian at heart, I still use frequentists hypothesis tests on a regular basis (at least until journal reviewers are comfortable with the Bayesain alternatives).

-

If I'm understanding you correctly, you're in agreement with the late, great Paul Meehl. See

Meehl, P.E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34:103-115.

-
Would you mind developing this answer a bit? –  chl Aug 17 '12 at 21:28