In textbook statistical tests, we usually calculate the probability of observing the data we observed given that the null hypothesis is true, i.e. $P[D|H_0]$. If this probability is small (e.g. $<0.05$), we claim that the null hypothesis is unlikely given the data, i.e. we reject the null hypothesis. That is, we claim that because $P[D|H_0]$ is small, $P[H_0|D]$ is also small. But that is not generally true.

As an example, this is similar to the following line of reasoning that leads to an incorrect conclusion: If a person is an American, he is probably not a member of Congress. This person is a member of Congress. Therefore, he is probably not an American. (Pollard & Richardson, 1987)

And yet, rejection of the null hypothesis based on small p-values appears to be both widely taught and widely used. Why? What are the assumptions on the Bayesian priors that allow us to reject the null hypothesis based on the low likelihood of the data under the null hypothesis?

  • 1
    $\begingroup$ Usually, in a hypothesis test the situation is designed so that $H_0$ and $H_1$ are contradictory and $H_1$ leads to the obtained data being (relatively) likely. The Americans/Congress example fails this: $H_1$ here is “the person is not an American”, which actually reduces the probability of the observed outcome (member of Congress). Has it perhaps been shown that under properly set up null and alternative hypothesis, the likelihood is close enough to the posterior under some sensible assumptions? $\endgroup$ – rinspy Sep 1 '17 at 9:29
  • 1
    $\begingroup$ I think the premise of the question is wrong (i.e. your characterization of how the argument goes is not a suitable nor a typical argument for hypothesis testing). I don't think either Fisher or Neyman and Pearson make the claim that if $P(D|H_0)$ is small, so is $P(H_0|D)$; indeed that reads like an argument a Bayesian would use to suggest there was a problem (which if it's not how the proponents make the argument, would be something of a straw man). Where does this argument come from? $\endgroup$ – Glen_b -Reinstate Monica Sep 1 '17 at 9:35
  • 2
    $\begingroup$ see stats.stackexchange.com/questions/163957/… $\endgroup$ – user83346 Sep 1 '17 at 11:54
  • 2
    $\begingroup$ "$P[H_0\mid D]$" has no meaning in the standard (non-Bayesian) theory of hypothesis testing. This is extensively explained in the thread I linked to. $\endgroup$ – whuber Sep 1 '17 at 15:48
  • 3
    $\begingroup$ The meaning is clear if you assume $H_0$ is random. That assumption is not part of the theory. Without that assumption, the expression "$P[H_0\mid D]$" is meaningless. Thus, no formal argument can possibly be made about it. $\endgroup$ – whuber Sep 1 '17 at 16:23