Which one is the null hypothesis? Conflict between science theory, logic and statistics?

Question

I am having difficulties understanding the underlying logic in setting the null hypothesis. In this answer the obviously generally accepted proposition is stated that the null hypothesis is the hypothesis that there will be no effect, everything stays the same, i.e. nothing new under the sun, so to speak.

The alternative hypothesis is then what you try to prove, that e.g. a new drug delivers on its promises.

Now coming form science theory and general logic we know that we can only falsify propositions, we cannot prove something (no number of white swans can prove that all swans are white but one black swan can disprove it). This is why we try to disprove the null hypothesis, which is not equivalent to proving the alternative hypothesis - and this is where my skepticism starts - I will give an easy example:

Let's say I want to find out what kind of animal is behind a curtain. Unfortunately I cannot directly observe the animal but I have a test which gives me the number of legs of this animal. Now I have the following logical reasoning:

If the animal is a dog then it will have 4 legs.

If I conduct the test and find out that it has 4 legs this is no proof that it is a dog (it can be a horse, a rhino or any other 4-legged animal). But if I find out that it has not 4 legs this is a definite proof that it can not be a dog (assuming a healthy animal).

Translated into drug effectiveness I want to find out if the drug behind the curtain is effective. The only thing I will get is a number that gives me the effect. If the effect is positive, nothing is proved (4 legs). If there is no effect, I disprove the effectiveness of the drug.

Saying all this I think - contrary to common wisdom - the only valid null hypothesis must be

The drug is effective (i.e.: if the drug is effective you will see an effect).

because this is the only thing that I can disprove - up to the next round where I try to be more specific and so on. So it is the null hypothesis that states the effect and the alternative hypothesis is the default (no effect).

Why is it that statistical tests seem to have it backwards?

P.S.: You cannot even negate the above hypothesis to get a valid equivalent hypothesis, so you cannot say "The drug is not effective" as a null hypothesis because the only logically equivalent form would be "if you see no effect the drug will not be effective" which brings you nowhere because now the conclusion is what you want to find out!

P.P.S.: Just for clarification after reading the answers so far: If you accept scientific theory, that you can only falsify statements but not prove them, the only thing that is logically consistent is choosing the null hypothesis as the new theory - which can then be falsified. Because if you falsify the status quo you are left empty handed (the status quo is disproved but the new theory far from being proved!). And if you fail to falsify it you are in no better position either.

Answer 1

In statistics there are tests of equivalence as well as the more common test the Null and decide if sufficient evidence against it. The equivalence test turn this on its head and posits that effects are different as the Null and we determine if there is sufficient evidence against this Null.

I'm not clear on your drug example. If the response is a value/indicator of the effect, then an effect of 0 would indicate not effective. One would set that as the Null and evaluate the evidence against this. If the effect is sufficiently different from zero we would conclude that the no-effectiveness hypothesis is inconsistent with the data. A two-tailed test would count sufficiently negative values of effect as evidence against the Null. A one tailed test, the effect is positive and sufficiently different from zero, might be a more interesting test.

If you want to test if the effect is 0, then we'd need to flip this around and use an equivalence test where the H0 is the effect is not equal to zero, and the alternative is that H1 = the effect = 0. That would evaluate the evidence against the idea that effect was different from 0.

Answer 2

I think this is another case where frequentist statistics can't give a direct answer to the question you actually want to ask, and so answers a (no so) subtly different question, and it is easy to misinterpret this as a direct answer to the question you actually wanted to ask.

What we would really like to ask is normally what is the probability that the alternative hypothesis is true (or perhaps how much more likely to be true is it than the null hypothesis). However a frequentist analysis fundamentally cannot answer this question, as to a frequentist a probability is a long run frequency, and in this case we are interested in the truth of a particular hypothesis, which doesn't have a long run frequency - it is either true or it isn't. A Bayesian on the other hand can answer this question directly, as to a a Bayesian a probability is a measure of the plausibility of some proposition, so it is perfectly reasonable in a Bayesian analysis to assign a probability to the truth of a particular hypothesis.

The way frequentists deal will particular events is to treat them as a sample from some (possibly fictitious) population and make a statement about that population in place of a statement about the particular sample. For example, if you want to know the probability that a particular coin is biased, after observing N flips and observing h heads and t tails, a frequentist analysis cannot answer that question, however they could tell you the proportion of coins from a distribution of unbiased coins that would give h or more heads when flipped N times. As the natural definition of a probability that we use in everyday life is generally a Bayesian one, rather than a frequentist one, it is all too easy to treat this as the pobability that the null hypothesis (the coin is unbiased) is true.

Essentially frequentist hypothesis tests have an implicit subjectivist Bayesian component lurking at its heart. The frequentist test can tell you the likelihood of observing a statistic at least as extreme under the null hypothesis, however the decision to reject the null hypothesis on those grounds is entirely subjective, there is no rational requirement for you to do so. Essentiall experience has shown that we are generally on reasonably solid ground to reject the null if the p-value is suffciently small (again the threshold is subjective), so that is the tradition. AFAICS it doesn't fit well into the philosophy or theory of science, it is essentially a heuristic.

That doesn't mean it is a bad thing though, despite its imperfections frequentist hypothesis testing provides a hurdle that our research must get over, which helps us as scientists to keep our self-skepticism and not get carried away with enthusiasm for our theories. So while I am a Bayesian at heart, I still use frequentists hypothesis tests on a regular basis (at least until journal reviewers are comfortable with the Bayesain alternatives).

Answer 3

To add to Gavin's answer, a couple of things:

First, I've heard this idea that propositions can only be falsified, but never proven. Could you post a link to a discussion of this, because with our wording here it doesn't seem to hold up very well - if X is a proposition, then not(X) is a proposition too. If disproving propositions is possible, then disproving X is the same as proving not(X), and we've proven a proposition.

Second, your analogy between the P(effective|$test_+$) and P(dog|4 legs) is interesting. The wording should be changed a little bit though:

The drug is effective (i.e.: iff the drug is effective you will see an effect).

In fact, P(effective|$test_+$) is often greater than P($test_+$|effective), as long as you use hypothesis testing and the right statistical model. Hypothesis testing formalizes the unlikelihood of positive test results under $H_0$. But an effective drug doesn't guarentee a positive test; when the drug is effective and variance is high the effect can be masked in the test.

If you observe $test_+$ you can infer effectiveness, because the alternative is $H_0$, and the hypothesis testing is set up so that P($test_+$|$H_0$) < 0.05.

So the difference between the dog case and the effectiveness case is in the appropriateness of the inference from the evidence to the conclusion. In the dog case, you have observed some evidence that doesn't strongly imply a dog. But in the clinical trial case you have observed some evidence that does strongly imply efficacy.

Answer 4

You are right that, in a sense, frequentist hypothesis testing has it backwards. I'm not saying that that approach is wrong, but rather that the results are often not designed to answer the questions that the researcher is most interested in. If you want a technique more similar to the scientific method, try Bayesian inference.

Instead of talking about a "null hypothesis" that you can reject or fail to reject, with Bayesian inference you begin with a prior probability distribution based upon your understanding of the situation at hand. When you acquire new evidence, Bayesian inference provides a framework for you to update your belief with the evidence taken into account. I think this is how more similar to how science works.

Answer 5

I think you've got a fundamental error here (not that the whole area of hypothesis testing is clear!) but you say the alternative is what we try to prove. But this is not right. We attempt to reject (falsify) the null. If the results we obtain would be very unlikely if the null were true, we reject the null.

Now, as others said, this is not usually the question we want to ask: We don't usually care how likely the results are if the null is true, we care how likely the null is, given the results.

Answer 6

I'll expand on the mention of Paul Meehl by @Doc:

1) Testing the opposite of your research hypothesis as the null hypothesis makes it so you can only affirm the consequent which is a "formally invalid" argument. The conclusions do not necessarily follow from the premise.

If Bill Gates owns Fort Knox, then he is rich.
Bill Gates is rich.
Therefore, Bill Gates owns Fort Knox.

http://rationalwiki.org/wiki/Affirming_the_consequent

If the theory is "This drug will improve recovery" and you observe improved recovery this does not mean you can say your theory is true. The appearance of improved recovery could have occurred for some other reason. No two groups of patients or animals will be exactly the same at baseline and will change further over time during the study. This is a greater problem for observational than experimental research because randomization "defends" against severe imbalances of unknown confounding factors at baseline. However, randomization does not really resolve the problem. If the confounds are unknown we have no way to tell the extent to which the "randomization defense" has been successful.

Also see table 14.1 and the discussion of why no theory can be tested on it's own (there are always auxiliary factors that tag along) in:

Paul Meehl. "The Problem Is Epistemology, Not Statistics: Replace Significance Tests by Confidence Intervals and Quantify Accuracy of Risky Numerical Predictions" In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What If There Were No Significance Tests? (pp. 393–425) Mahwah, NJ : Erlbaum, 1997.

2) If some type of bias is introduced (e.g., imbalance on some confounding factors) we do not know which direction this bias will lie or how strong it is. The best guess we can give is that there is a 50% chance of biasing the treatment group in the direction of higher recovery. As sample sizes get large there is also 50% chance that your significance test will detect this difference and you will interpret the data as corroborating your theory.

This situation is totally different from the case of a null hypothesis that "This drug will improve recovery by x%". In this case the presence of any bias (which I would say always exist in comparing groups of animals and humans) makes it more likely for you to reject your theory.

Think of the "space" (Meehl calls it the "Spielraum") of possible results bounded by the most extreme measurements possible. Perhaps there can be 0-100% recovery, and you can measure with resolution of 1%. In the common significance testing case, the space consistent with your theory will be 99% of the possible outcomes you could observe. In the case when you predict a specific difference the space consistent with your theory will be 1% of the possible outcomes.

Another way of putting it is that finding evidence against a null hypothesis of mean1=mean2 is not a severe test of the research hypothesis that a drug does something. A null of mean1 < mean2 is better but still not very good.

See figure 3 and 4 here: (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant using it. Psychological Inquiry, 1, 108-141, 173-180

Answer 7

If I'm understanding you correctly, you're in agreement with the late, great Paul Meehl. See

Meehl, P.E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34:103-115.

Answer 8

Isn't all statistics premised on the assumption that nothing is certain in the natural world (as distinct from the man-made world of games &c). In other words, the only way we can get near to understanding it is by measuring the probability that one thing correlates with another and this varies between 0 and 1 but can only be 1 if we could test the hypothesis an infinite number of times in an infinite number of different circumstances, which of course is impossible. And we can never know it was zero for the same reason. It's a more reliable approach to understanding the reality of nature, than mathematics, which deal in absolutes and mostly relies on equations, which we know are idealistic because if, literally, the LH side of an equation really = the RH side, the two sides could be reversed and we wouldn't learn anything. Strictly speaking it applies only to a static world, not a 'natural' one which is intrinsically turbulent. Hence, the null hypothesis should even underwrite mathematics - whenever it is used to understand nature itself.

Answer 9

I think the problem is in the word 'true'. The reality of the natural world is innately un-knowable as it's infinitely complex and infinitely variable over time, so 'truth' applied to nature is always conditional. All we can do is try to find levels of probable correspondence between variables by repeated experiment. In our attempt to make sense of reality, we look for what seems like order in it and construct conceptually-conscious models of it in our mind to help us make sensible decisions BUT it's very much a hit-and-miss affair because there's always the unexpected. The null hypothesis is the only reliable starting point in our attempt to make sense of reality.

Answer 10

We must select null hypothesis the one which we want to reject.

Because in our hypothesis testing scenario, there is a critical region, if the region under hypothesis come in critical region , we reject the hypothesis otherwise we accept the hypothesis.

So suppose we select the null hypothesis , the one we want to accept. And the region under null hypothesis does not come under critical region, So we will accept the null hypothesis. But the problem here is if region under null hypothesis come under acceptable region, then it does not mean that the region under alternate hypothesis will not come under acceptable region. And if this is the case then our interpretation about result will be wrong. So we must only take that hypothesis as a null hypothesis which we want to reject. If we are able to reject null hypothesis, then it means that alternate hypothesis is true. But if we are not able to reject null hypothesis, then it means that any of the two hypothesis can be correct. May be we can then take another test, in which we can can take our alternate hypothesis as null hypothesis, and then we can attempt to reject it. If we are able to reject the alternate hypothesis(which now is null hypothesis.) , then we can say that our initial null hypothesis was true.