Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Is there a difference between the phrases "testing of hypothesis" and "test of significance" or are they the same?

After a detailed answer from @Micheal Lew, I have one confusion that nowadays hypothesis (e.g., t-test to test mean) are example of either "significance testing" or "hypothesis testing"? Or it is a combination of both? How you would differentiate them with simple example?

share|improve this question
1  
Student's t-test can be used to provide a p value that can then be used in a Fisherian signifcance test (the p value is the level of significance) or in a Neyman-pearsonian hypothesis test (if the p value is less than the preset alpha then the result is 'significant'). The differemce is in what is done with the t-test result rather than which school of thought the t-test comes from (although Gossett's approach had much more in common with Fisher than with N-P). –  Michael Lew Sep 30 '11 at 19:40
    
Thanks, it's clear now. –  love-stats Oct 1 '11 at 4:11

2 Answers 2

up vote 14 down vote accepted

Significance testing is what Fisher devised and hypothesis tesing is what Neyman and Pearson devised to replace significance testing. They are not the same and are mutually incompatible to an extent that would surprise most users of null hypothesis tests.

Fisher's significance tests yield a p value that represents how extreme the observations are under the null hypothesis. That p value is an index of evidence against the null hypothsis and the level of significance.

Neyman and Pearson's hypothesis tests set up both a null hypothesis and an alternative hypothesis and work as a decision rule for accepting the null hypothesis. Briefly (there is more to it than I can put here) you choose an acceptable rate of false positive inference, alpha (usually 0.05), and either accept or reject the null based on whether the p value is above or below alpha. You have to abide by the statistical test's decision if you wish to protect against false positive errors.

Fisher's approach allows you to take anything you like into account in interpreting the result, for example pre-existing evidence can be informally taken into account in the interpretation and presentation of the result. In the N-P approach that can only be done in the experimental design stage, and seems to be rarely done. In my opinion the Fisherian approach is more useful in basic bioscientific work than is the N-P approach.

There is a substantial literature about inconsistencies between significance testing and hypothesis testing and about the unfortunate hybridisation of the two. You could start with this paper: Goodman, Toward evidence-based medical statistics. 1: The P value fallacy. http://www.ncbi.nlm.nih.gov/sites/entrez?Db=pubmed&Cmd=ShowDetailView&TermToSearch=10383371

share|improve this answer
1  
@Micheal Lew - +1 i wasn't aware that Newman/Pearson had coined the phrase hypothesis testing, and I interpreted it in a rather more informal fashion. Also, can you please elaborate on how my answer is wrong, as I would like to correct any errors and am always eager to hear feedback. –  richiemorrisroe Sep 29 '11 at 14:29
    
@richiemorrisroe - Neyman and Pearson did more than coin a phrase! They devised a whole paradigm for statistical analysis--a paradigm that predominates in many areas today (despite my opinion, and Fisher's) that it is ill-suited to most scientific experimentation. Fisher claimed repeatedly that the N-P approach was only relevant to industrial acceptance testing. Most introductory statistics texts fail to include enough detail and history to allow students to understand that there are important differences between the schools of thought about statistical testing. It's unfortunate. –  Michael Lew Sep 29 '11 at 17:10

In many cases, these two statements mean the same thing. However, they can also be quite different.

Testing a hypothesis consists of first saying what you believe will occur with some phenomenon, then developing some kind of test for this phenomenon, and then determining whether or not the phenomenon actually occurred. In many cases, testing of a hypothesis need not involve any kind of statistical test. I am reminded of this quote by the physicist Ernest Rutherford - If your experiment needs statistics, you ought to have done a better experiment. That being said, testing of hypotheses normally does use some kind of statistical tool.

In contrast, testing of significance is a purely statistical concept. In essence, one has two hypotheses - the null hypothesis, which states that there is no difference between your two (or more) collections of data. The alternative hypothesis is that there is a difference between your two samples that did not occur by chance.

Based on the design of your study, you then compare the two (or more) samples using a statistical test, which gives you a number, which you then compare to a reference distribution (like the normal, t, or F distributions) and if this test statistic exceeds a critical value, you reject the null hypothesis and conclude that there is a difference between the two (or more) samples. This criterion is normally that the probability of the difference occurring by chance is less than one in twenty (p<0.05), though others are sometimes used.

share|improve this answer
    
Could you please give some example where hypothesis testing do not involve any kind of statistical tests? –  love-stats Sep 29 '11 at 13:24
    
This is an inaccurate representation of significance testing and hypothesis testing. –  Michael Lew Sep 29 '11 at 13:57
    
@user152509 suppose i conduct a study in which I interview users and non users of a particular product. I hypothesise that non-users will focus on the disadvantages of said product, while users will talk about how the product helps them. This is what I observe, hence a hypothesis tested without statistics. –  richiemorrisroe Sep 29 '11 at 14:26
1  
It is important to discriminate between a scientific hypothesis and a statistical hypothesis. The null hypothesis tested by null hypothesis statistical tests is usually only e latter. Testing a well-designed statistical hypothesis may allow inference regarding the scientific hypothsis, but it is not always the case. –  Michael Lew Sep 29 '11 at 17:03
    
@Micheal Lew, I have one confusion that nowadays hypothesis (e.g., t-test to test mean) are example of either "significance testing" or "hypothesis testing"? Or it is a combination of both? How you would differentiate them with simple example? –  love-stats Sep 29 '11 at 22:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.