Can you create an example showing when the p-value $P(D|H_0)$ does not imply probability of $H_0$ being true given the observed data $P(H_0|D)$?

Question

I'm reading The Insignificance of Null Hypothesis Significance Testing and on page 654, the author states that most people incorrectly think that the null hypothesis significance test produces $\mathbb{P}(H_0|D)$: the probability of $H_0$ being true given the observed data.

But the test actually produces $\mathbb{P}(D|H_0)$. And by Bayes law, these two are not the same unless $\mathbb{P}(H_0) = \mathbb{P}(D)$. What is the intuitive meaning of this result?

Can someone give me an example where using typical hypothesis i.e. $H_0: \mu = 0$ and $H_1: \mu \ne 0$, $\bar{X}$ (sample mean) test statistic and the normal distribution, where hypothesis tests start to fail?

Answer 1

Is not possible to create the example that you looking for. The problem here is that $P(H_0|D)$ is a meaningless writing. The last because, even if in practice case you do not known if $H_0$ is true or false, we have to remember that, under the paradigm behind the p-value (ML and/or LS and then frequentist approach) parameters are unknown constants and not random variables.

Then in your example $\mu$ is a fixed constant (known or unknown) and something like $P(-x<\mu<x)$ is a nonsense writing. As a consequence even $P(H_0|D)$ is senseless. At the other side is possible to shown that p-value=$P(D|H_0)$ make sense.

Therefore, even if it seems an intuitive way, is not possible to use the bayes rule here in order to link $P(D|H_0)$ and $P(H_0|D)$

Therefore

But the test actually produces $P(D|H_0)$. And by Bayes law, these two are not the same unless $P(H_0)=P(D)$. What is the intuitive meaning of this result?

It haven't proper meaning.