1
$\begingroup$

I am trying to find an intuitive example to the following situation (to teach my social scientist wife):

When we set alpha to be small, we are reducing the probability of rejecting the null hypothesis when it is true. However, when we reject the null hypothesis at an alpha level, it does not mean that "the alternative hypothesis was actually true with a probability of alpha." The null hypothesis is either true or false, and we cannot do anything about it.

She responded "what is the point of conducting statistical tests if we cannot find the probability whether H0 is true of false, given that we found a significant result."

So, I am wondering if there is a nice example that can demonstrate the incorrectness of "the probability of H0 is true given a significant result is alpha."

enter image description here

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ I cannot completely follow what you are looking for, but one example comes to mind that might help you clarify some issues: suppose your null hypothesis is that half of a population would answer "yes" to a particular question -- and suppose the population consists of an odd number of people. You can readily think of opinion polls in which the p-value will exceed $0.05$ and others where it will be less than $0.05,$ but the null hypothesis itself is mathematically impossible regardless! $\endgroup$
    – whuber ♦
    Dec 1, 2020 at 20:42
  • $\begingroup$ @whuber Interesting counterexample, but that seems more like an issue between sample and population rather than hypothesis testing itself - there really is no need to perform a hypothesis test in the first place if you have the whole population (you can observe everything, there's nothing left to infer). If I do a coin-flipping experiment, for example, the null hypothesis of a fair coin doesn't become impossible simply by choosing to conduct an odd number of flips. $\endgroup$
    – Nuclear Hoagie
    Dec 1, 2020 at 21:20
  • $\begingroup$ @Nuclear I haven't assumed anything about the population except its count. $\endgroup$
    – whuber ♦
    Dec 1, 2020 at 21:39
  • 1
    $\begingroup$ This is my usual way of explaining this using small children. $\endgroup$
    – dimitriy
    Dec 1, 2020 at 22:05
  • $\begingroup$ @DimitriyV.Masterov the toothbrush example is perfect, thank you. $\endgroup$
    – ThePortakal
    Dec 1, 2020 at 22:18

1 Answer 1

Reset to default
0
$\begingroup$

Maybe the case of Paul the psychic octopus?

Summary: this octopus's predictions of football match winners were correct on 12 out of 14 occasions, which is statistically significant at the 95% level. But the probability that the octopus was able to predict results obviously is less than 95% - he probably just got lucky repeatedly.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.