I am trying to find an intuitive example to the following situation (to teach my social scientist wife):
When we set alpha to be small, we are reducing the probability of rejecting the null hypothesis when it is true. However, when we reject the null hypothesis at an alpha level, it does not mean that "the alternative hypothesis was actually true with a probability of alpha." The null hypothesis is either true or false, and we cannot do anything about it.
She responded "what is the point of conducting statistical tests if we cannot find the probability whether H0 is true of false, given that we found a significant result."
So, I am wondering if there is a nice example that can demonstrate the incorrectness of "the probability of H0 is true given a significant result is alpha."