# What is the difference between "testing of hypothesis" and "test of significance"?

Is there a difference between the phrases "testing of hypothesis" and "test of significance" or are they the same?

After a detailed answer from @Micheal Lew, I have one confusion that nowadays hypothesis (e.g., t-test to test mean) are example of either "significance testing" or "hypothesis testing"? Or it is a combination of both? How you would differentiate them with simple example?

• Student's t-test can be used to provide a p value that can then be used in a Fisherian signifcance test (the p value is the level of significance) or in a Neyman-pearsonian hypothesis test (if the p value is less than the preset alpha then the result is 'significant'). The differemce is in what is done with the t-test result rather than which school of thought the t-test comes from (although Gossett's approach had much more in common with Fisher than with N-P). Sep 30, 2011 at 19:40

Significance testing is what Fisher devised and hypothesis testing is what Neyman and Pearson devised to replace significance testing. They are not the same and are mutually incompatible to an extent that would surprise most users of null hypothesis tests.

Fisher's significance tests yield a p value that represents how extreme the observations are under the null hypothesis. That p value is an index of evidence against the null hypothesis and is the level of significance.

Neyman and Pearson's hypothesis tests set up both a null hypothesis and an alternative hypothesis and work as a decision rule for accepting the null hypothesis. Briefly (there is more to it than I can put here) you choose an acceptable rate of false positive inference, alpha (usually 0.05), and either accept or reject the null based on whether the p value is above or below alpha. You have to abide by the statistical test's decision if you wish to protect against false positive errors.

Fisher's approach allows you to take anything you like into account in interpreting the result, for example pre-existing evidence can be informally taken into account in the interpretation and presentation of the result. In the N-P approach that can only be done in the experimental design stage, and seems to be rarely done. In my opinion the Fisherian approach is more useful in basic bioscientific work than is the N-P approach.

There is a substantial literature about inconsistencies between significance testing and hypothesis testing and about the unfortunate hybridisation of the two. You could start with this paper: Goodman, Toward evidence-based medical statistics. 1: The P value fallacy. https://pubmed.ncbi.nlm.nih.gov/10383371/

• @Micheal Lew - +1 i wasn't aware that Newman/Pearson had coined the phrase hypothesis testing, and I interpreted it in a rather more informal fashion. Also, can you please elaborate on how my answer is wrong, as I would like to correct any errors and am always eager to hear feedback. Sep 29, 2011 at 14:29
• @richiemorrisroe - Neyman and Pearson did more than coin a phrase! They devised a whole paradigm for statistical analysis--a paradigm that predominates in many areas today (despite my opinion, and Fisher's) that it is ill-suited to most scientific experimentation. Fisher claimed repeatedly that the N-P approach was only relevant to industrial acceptance testing. Most introductory statistics texts fail to include enough detail and history to allow students to understand that there are important differences between the schools of thought about statistical testing. It's unfortunate. Sep 29, 2011 at 17:10

In many cases, these two statements mean the same thing. However, they can also be quite different.

Testing a hypothesis consists of first saying what you believe will occur with some phenomenon, then developing some kind of test for this phenomenon, and then determining whether or not the phenomenon actually occurred. In many cases, testing of a hypothesis need not involve any kind of statistical test. I am reminded of this quote by the physicist Ernest Rutherford - If your experiment needs statistics, you ought to have done a better experiment. That being said, testing of hypotheses normally does use some kind of statistical tool.

In contrast, testing of significance is a purely statistical concept. In essence, one has two hypotheses - the null hypothesis, which states that there is no difference between your two (or more) collections of data. The alternative hypothesis is that there is a difference between your two samples that did not occur by chance.

Based on the design of your study, you then compare the two (or more) samples using a statistical test, which gives you a number, which you then compare to a reference distribution (like the normal, t, or F distributions) and if this test statistic exceeds a critical value, you reject the null hypothesis and conclude that there is a difference between the two (or more) samples. This criterion is normally that the probability of the difference occurring by chance is less than one in twenty (p<0.05), though others are sometimes used.

• Could you please give some example where hypothesis testing do not involve any kind of statistical tests? Sep 29, 2011 at 13:24
• This is an inaccurate representation of significance testing and hypothesis testing. Sep 29, 2011 at 13:57
• @user152509 suppose i conduct a study in which I interview users and non users of a particular product. I hypothesise that non-users will focus on the disadvantages of said product, while users will talk about how the product helps them. This is what I observe, hence a hypothesis tested without statistics. Sep 29, 2011 at 14:26
• It is important to discriminate between a scientific hypothesis and a statistical hypothesis. The null hypothesis tested by null hypothesis statistical tests is usually only e latter. Testing a well-designed statistical hypothesis may allow inference regarding the scientific hypothsis, but it is not always the case. Sep 29, 2011 at 17:03
• @Micheal Lew, I have one confusion that nowadays hypothesis (e.g., t-test to test mean) are example of either "significance testing" or "hypothesis testing"? Or it is a combination of both? How you would differentiate them with simple example? Sep 29, 2011 at 22:37