Reading the answers to this thread, I started wondering about how Hypothesis Testing relates to the Scientific Method. While I have a good understanding of both, I am having a hard time drawing the precise connection between them.

At a high level, the scientific method, comes down to:

  • Make conjectures & hypotheses (theory)
  • Make predictions from this theory
  • Carry out experiments and observations
  • Test and embrace the new theory if

    • the data fit the predictions (more) accurately than alternative theories
    • the new theory is not more complex than other plausible alternatives

At a high level, it looks to me that the scientific method thus follows an "accept-if-fits-well" approach which contrasts with the "reject-if-it-doesn't-fit" approach from statistical hypothesis testing. Is this correct? and if so, why is this the case? Aren't they both fundamentally chasing the same goal; inferring the theory or model that best explains the observations?

  • 1
    $\begingroup$ The new theory doesn't have to be simpler than alternative theories. Also, another feature of new theories is that they usually encompass the old theories. E.g. special relativity covers Newton's motion theory. Maxwell's equations cover Ohm's law etc. $\endgroup$ – Aksakal Dec 30 '14 at 14:55
  • 1
    $\begingroup$ But what do people usually use NHST to reject? Usually it is not their own or anyone else's hypothesis. Hypothesis testing is fine if you have a theory/hypothesis to test with it. $\endgroup$ – Livid Dec 30 '14 at 15:01
  • 3
    $\begingroup$ This characterization of the scientific method does not appear to conform to what scientists actually do, nor to how philosophers write about it. In particular, I am not aware of anyone who has articulated or advocated an "accept-if-fits approach": this sounds almost the opposite of the scientific method, which would be much better (if oversimplistically) characterized as "reject-if-doesn't fit." But maybe I misunderstand: would you have a reference to support your characterization? $\endgroup$ – whuber Dec 30 '14 at 16:12
  • 2
    $\begingroup$ @whuber, almost everything in my post I got from the definition on Wikipedia for the scientific method. The "accept-if-fits-well" vs "reject-if-it-doesn't-fit" is my own characterization to summarize the question. If this characterization is incorrect, an explanation of why that's the case would constitute, IMHO, an answer. I will rephrase the OP to make that clear. $\endgroup$ – Amelio Vazquez-Reina Dec 30 '14 at 16:16
  • 4
    $\begingroup$ @Aksakal I don't think that is correct. You can only disprove the conjunction of the theory + various assumptions (eg, the equipment was functioning correctly). People will not throw out a theory that seems useful just due to some conflicting evidence. I think almost always people do not believe the theory is 100% correct anyway, the theory needs to be replaced by something more convincing. I'd be interested if you have an example in mind though. $\endgroup$ – Livid Dec 30 '14 at 16:31

These issues have been known for a long time, it started in education research, psychology and has since spread to even physics. There is no one in particular to blame and apparently nothing can stop it.

We are quite in danger of sending highly trained and highly intelligent young men out into the world with tables of erroneous numbers under their arms, and with a dense fog in the place where their brains ought to be. In this century, of course, they will be working on guided missiles and advising the medical profession on the control of disease, and there is no limit to the extent to which they could impede every sort of national effort.

Fisher, R N (1958). "The Nature of Probability". Centennial Review 2: 261–274.

The usual application of statistics in psychology consists of testing a "null hypothesis" that the investigator hopes is false. For example, he tests the hypothesis that the ex perimental group is the same as the control group even though he has done his best to make them perform differently.Then a "significant" difference is obtained which shows that the data do not agree with the hypothesis tested. The experimenter is then pleased because he has shown that a hypothesis he didn't believe, isn't true. Having found a "significant difference," the more important next step should not be neglected. Namely, formulate a hypothesis that the scientist does believe and show that the data do not differ significantly from it. This is an indica tion that the newer hypothesis may be regarded as true. A definite scientific advance has been achieved.


The major point of this paper is that the test of significance does not provide the information concerning psychological phenomena characteristically attributed to it; and that, furthermore, a great deal of mischief has been associated with its use. What will be said in this paper is hardly original. It is, in a certain sense, what "everybody knows." To say it "out loud" is, as it were, to assume the role of the child who pointed out that the emperor was really outfitted only in his underwear. Little of that which is contained in this paper is not already available in the literature, and the literature will be cited.


The puzzle, sufficiently striking (when clearly discerned) to be entitled to the designation “paradox,” is the follow- ing: In the physical sciences, the usual result of an improvement in experimental design, instrumentation, or numerical mass of data, is to increase the difficulty of the “observational hurdle” which the physical theory of interest must successfully surmount; whereas, in psychology and some of the allied behavior sciences, the usual effect of such improvement in experimental precision is to provide an easier hurdle for the theory to surmount. Hence what we would normally think of as improve- ments in our experimental method tend (when predictions materialize) to yieldstronger corroboration of the theory in physics, since to remain unrefuted the theory must have survived a more difficult test; by contrast, such experimental improvement in psychology typically results in a weaker corroboration of the theory, since it has now been required to survive a more lenient test.


  • 3
    $\begingroup$ How much of this represents a problem with the scientific method itself vs a problem of malpractice or improper use of the scientific method? $\endgroup$ – Amelio Vazquez-Reina Dec 30 '14 at 15:46
  • 1
    $\begingroup$ @user023472. I don't follow you. Those quotes are from people complaining about the replacement of science as it used to be practiced with something else. $\endgroup$ – Livid Dec 30 '14 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.