I don't understand three important issues :

(1) Why do we calculate the test statistic under the null hypothesis ?

(2) Why do we reject the null hypothesis when :

p-value < significance level ?

(3) Why is type-I error more serious than type-II error ?

  • $\begingroup$ To whoever downvoted I think this is a very good question. It was only a few years ago that I was taking statistics courses (for non-statisticians) and remember being confused about each of these issues due to how the material was presented. $\endgroup$ – Flask Dec 4 '13 at 16:52
  • 2
    $\begingroup$ I didn't downvote this, but I can easily impute objections to it as it stands. First, there is little evidence here of research effort on matters covered by hundreds if not thousands of texts. Second, it is potentially too broad to be useful. That said, the question is, as it were, justified by the answers to date. $\endgroup$ – Nick Cox Dec 4 '13 at 17:13


I found this pre-print paper by @Michael Lew to clarify many things for me. In terms of calculating a p-value under the null hypothesis, it can be seen as more of a matter of convenience than anything else:

the null hypothesis serves as little more than an anchor for the calculation|a landmark in parameter space

To P or not to P: on the evidential nature of P-values and their place in scientific inference


The original use of "significance" by Fisher grouped results into interesting/not interesting. If the p-value was not less than some threshold (unique to the given experiment determined subjectively) he would attribute it to low signal to noise ratio and ignore the result. There was no concept of type II error as one would never accept a hypothesis.

The Neyman-Pearson hypothesis testing framework instead determines the significance level via cost benefit analysis (what are the relative costs of Type I and Type II errors) using two (or more) plausible alternative hypotheses. The two approaches were combined into one by textbook writers starting in the 1940s and many claim the result makes no sense.

The wikipedia page on NHST needs some cleaning up to incorporate the contributions of different editors but it has many good references. Because the method that is taught is a result of historical accident (not reasoning or logic) it will be difficult for you to understand without knowledge of the history.


If we look at Fisher's early (first?) description of the significance test we see that in the case of the lady tasting tea it was plausible that she could not distinguish the order in which tea/milk was added at all. Thus this was a logical null hypothesis. He also states the significance level is "open to the experimenter".

However later he states that a null hypothesis that the death rates of two groups of animals are equal is also valid. In this case I think he simply missed the logical consequences of choosing an (often implausible; it is highly unlikely two death rates will be exactly the same) null hypothesis that is the opposite of the research hypothesis (why do the study unless you suspect the death rates will be different). These issues are described by Paul Meehl in Theory-Testing in Psychology and Physics: A Methodological Paradox and Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles That Warrant It

Quoting Fisher:

A LADY declares that by tasting a cup of tea made with milk she can discriminate whether the milk or the tea infusion was first added to the cup. We will consider the problem of designing an experiment by means of which this assertion can be tested.


It is open to the experimenter to be more or less exacting in respect to the smallness of the probability he would require before he would be willing to admit that his observations have demonstrated a positive result. It is obvious that an experiment would be useless of which no possible result would satisfy him. Thus, if he wishes to ignore results having probabilities as high as 1 in 20 -the probabilities being of course reckoned from the hypothesis that the phenomenon to be demonstrated is in fact absent- then it would be useless for him to experiment with only 3 cups of tea of each kind. For 3 objects can be chosen out of 6 in only 20 ways, and therefore complete success in the test would be achieved without sensory discrimination, i.e., by “pure chance,” in an average of 5 trials out of 100. It is usual and convenient for experimenters to take 5 per cent. as a standard level of significance, in the sense that they are prepared to ignore all results which fail to reach this standard, and, by this means, to eliminate from further discussion the greater part of the fluctuations which chance causes have introduced into their experimental results. No such selection can eliminate the whole of the possible effects of chance coincidence, and if we accept this convenient convention, and agree that an event which would occur by chance only once in 70 trials is decidedly “significant” in the statistical sense, we thereby admit that no isolated experiment, however significant in itself, can suffice for the experimental demonstration of any natural phenomenon; for the “one chance in a million” will undoubtedly occur, with no less and no more than its appropriate frequency, however surprised we may be that it should occur to us. In order to assert that a natural phenomenon is experimentally demonstrable we need, not an isolated record, but a reliable method of procedure. In relation to the test of significance, we may say that a phenomenon is experimentally demonstrable when we know how to conduct an experiment which will rarely fail to give us a statistically significant result.


It is evident that the null hypothesis must be exact, that is free from vagueness and ambiguity, because it must supply the basis of the “problem of distribution,” of which the test of significance is the solution. A null hypothesis may, indeed, contain arbitrary elements, and in more complicated cases often does so: as, for example, if it should assert that the death-rates of two groups of animals are equal, without specifying what these death-rates usually are. In such cases it is evidently the equality rather than any particular values of the death-rates that the experiment is designed to test, and possibly to disprove.

I think a crucial point that others have missed in reading Fisher is that he never specifies a need to specify the significance level beforehand. A reasonable level could be chosen given the experimental conditions that unfolded. Also no strong conclusion about a natural phenomenon should be derived from a single study anyway, a single study could only provide evidence pointing towards the phenomenon's existence, thus suggesting the usefulness of further study. Thus the significance level could be arbitrary and based on convenience (since it could change) but it was not "binding".

While for Neyman-Pearson a single study could not provide any evidence for the existence of a phenomenon. The "significance" level was definitely non-arbitrary but it was binding if the pre-specified error rate of the decision making process were to be valid in the long run. Citing Neyman and Pearson the same as @MichaelLew in the above arxiv paper:

We are inclined to think that as far as a particular hypothesis is concerned, no test based upon the theory of probability can by itself provide any valuable evidence of the truth or falsehood of that hypothesis.

But we may look at the purpose of tests from another view-point. Without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behaviour with regard to them, in following which we insure that, in the long run of experience, we shall not be too often wrong.

The hybrid approach (which is being taught to researchers) uses a pre-specified arbitrary and binding significance level which is different from both. It neither adjusts the level due to experimental conditions nor involves cost-benefit analysis.

  • $\begingroup$ I'd suggest that 1) is a little too loose. It's true that the null hypothesis gives a reference level, but it is just foolish to use a reference level that is not of interest or (a different but related matter) to use a significance test when the science is better served by something else (notably, a confidence interval). $\endgroup$ – Nick Cox Dec 4 '13 at 14:56
  • 2
    $\begingroup$ Although the question obliges you to simplify, drastically, your historical comments have to be taken with a pinch of salt. Fisher in practice often used, or appeared to use, fixed-level testing even while in broad discussions he also denounced it as naive or worse, while Neyman and Pearson's practice was also wider than their theoretical papers implied. I doubt that you can find Fisher writing of "signal-noise ratio". $\endgroup$ – Nick Cox Dec 4 '13 at 15:01
  • $\begingroup$ @NickCox I agree but would add that unless the null hypothesis is determined by theory or conventional wisdom (e.g., telepathy does not exist at all, good batches of beer have specific gravity X) It is always foolish to perform a significance test rather than do parameter estimation. $\endgroup$ – Flask Dec 4 '13 at 15:02
  • $\begingroup$ @NickCox see the middle bolded portion of the fisher quote where he speaks of "fluctuations". Also, I have seen mention of Pearson using significance testing in the Fisher sense but never seen this paper, if you have a source it would be appreciated. The applied work I have seen by was an attempt to distinguish between two plausible scientific hypotheses. $\endgroup$ – Flask Dec 4 '13 at 16:09
  • 1
    $\begingroup$ I guess we agree on all substantive and substantial points. It's a matter of trying hard to be (a) correct and (b) comprehensible to learners at the same time! Significance tests in moderately broad senses go back to the 18th century. On Fisher and Neyman as both saying one thing and often doing the opposite, see e.g. D.R. Cox. 2006. Principles of statistical inference. Cambridge U.P. p.43. I was just pointing out that the wording "signal-noise ratio" is anachronistic, but equivalent ideas can be found, no doubt. $\endgroup$ – Nick Cox Dec 4 '13 at 16:39
  1. The test statistic is chosen to be a measure of the discordance of the data with the null hypothesis, in some direction of interest (e.g. the difference of sample means between two groups, the correlation between successive observations in time, &c.). The bigger it gets, the more evidence against the null hypothesis.

  2. Well, we don't always. Sometimes the p-value is used just to calibrate the test statistic: a p-value of 0.03 means that you'd see a test statistic as big or bigger than the one you saw only 3 times out of a hundred if you were to keep repeating the measurement and if the null hypothesis were true. But if you want to say "I reject it" or "I don't reject it" (& to follow some different course of action in each case) then, when the null hypothesis is true, the p-value will be lower than a significance level 0.025 with probability 0.025 (it's uniformly distributed); thus you control Type I error, the probability of rejecting the null hypothesis when it is in fact true, to 0.025.

  3. More serious in what way? If you mean that making Type I errors has worse consequences, that's clearly not true in general.

† It's a bit different when you're sampling discrete variables—you'd have to say it'd be lower than the significance level with probability 0.025 or less.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.