Is the p-value equivalent to the false alarm value in the Bayesian rule?

Question

Is a p-value from a traditional significance test the same as the false alarm value in the Bayesian rule? And/or is it "close enough" to give correct results when used that way?

The definitions of the two terms seem to be talking about the same things, but I know it's easy to be tripped up by subtleties. Wikipedia says that a p-value is "the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true."

This page says the false error rate P(D|H') in the Bayesian rule is the probability of observing D if H' is true.

In both cases, you are talking about the probability of seeing data that says H when the reality is H'.

However, I see two possible problems with assuming that the two are equivalent: D in the definition of P(D|H') seems to refer to a single datum, while the definition of a p-value seems to refer to a range of values ("at least as extreme as"); and I'm not quite smart enough to figure out whether H' is equivalent to the null hypothesis. In all the simple Bayesian examples I've worked through it certainly seems to be, but I haven't yet found a definitive statement.

I also haven't found a definitive statement of how p-value and the false alarm value are related if they aren't the same, given that they're both saying at least loosely analogous things about data and hypotheses.

Answer 1

They're related in that selecting the critical value for p amounts to selecting a false alarm rate, under the assumption that the null hypothesis is true. The difference between the two is what is observed and known prior to the test.

It's conventional to denote the probability of making a type I error, i.e. a false positive, as $\alpha$; in the example you linked above, $\alpha$ is known from observations of previous tests. In other words, under the (null) hypothesis that a patient doesn't have cancer, $\alpha = 0.2$. In that example, the prior probability of a positive condition is also known, and is what ultimately allows for computing the odds of cancer.

In a traditional significance test, the idea is that the experimenter sets a critical value for p at $\alpha$, such that they only report false findings one in $\alpha^{-1}$ times. Because experimenters wish to avoid reporting false positives, they set their own false alarm rate under the assumption that their test hypothesis is incorrect. You can consider a test result of $p < \alpha$ equivalent to the positive test result D in the example.