Bayesian criticism of frequentist p-value

Question

I recently started reading the Bayesian criticism of the p-value and it seems that there is a lot of discussion around the fact that a frequentist approach is not that good when the Null Hypothesis is true.

For instance in this paper the authors write that "p-values overstate the evidence against the null [...] this does not have to do with type-I or type-II errors; it is an “independent” property of p-value."

To illustrate this point, the authors show that when the null is true, the p-value has a uniform distribution.

What I do not get is that even when the null is true, a frequentist approach, thanks to the Central Limit Theorem, is still able to construct confidence intervals that includes 0 (non-significance) at the appropriate $\alpha$ level.

I do not get why the fact that the p-value is uniform when the null is true shows that a frequentist approach is biased. And what does it mean "independent property of p-value"?

library(tidyverse)
library(broom)

n=1000
x = rnorm(n,100,30)
d = 0
y = x*d + rnorm(n,0,20)
df = data.frame(y,x)
plot(x,y)
abline(lm(y~x), col = 'red')

r = replicate(1000, sample_n(df, size = 50), simplify = F)
m = r %>% map(~ lm(y~x,data = .)) %>% map(tidy)

# Central Limit Theorem
bind_rows(.id = 'sample', m) %>% filter(term =='x') %>% ggplot(aes(estimate)) + facet_grid(~term) + geom_histogram()

s = bind_rows(.id = 'sample', m) %>% filter(term =='x')
s$false_positive = ifelse(s$p.value < 0.05, 1, 0)
prop.table(table(s$false_positive))

# uniform
hist(s$p.value, breaks = 50)

Answer 1

The point that the authors are trying to make is a subtle one: they see it as a failure of NHST that, as $n$ gets arbitrarily large, the $p$-value doesn't tend to 1. It's a bit surprising that this doesn't contain any discussion of equivalence testing. To me it's somewhat obvious and reasonable that the p-value maintains its uniform distribution when the null is true considering larger and larger $n$. Large $n$ means having sensitivity to detect smaller and smaller effects, while the false positive error rate remains fixed. So under the somewhat constrained setting of the null being exactly true, the behavior of the $p$-value distribution doesn't depend on $n$ at all.

1. NHST is, in my mind, desirable specifically because there's no way of declaring a null hypothesis to be true, as my experimental design is setup specifically to disprove it. A non-significant result may mean that my experiment was underpowered or the assumptions were wrong, so there are risks associated with accepting the null that I'd rather not incur.

2. We never actually believe that the null hypothesis is true. Typically failed designs arise because the truth is too close to the null to be detectable. Having too much data is kind of a bad thing in this case, rather there's a subtle art in designing a study to obtain only enough sample size so as to reject the null when a meaningful difference is present.

3. One can design a frequentist test that sequentially tests for differences (one or two tailed), and depending on a negative result, performs an equivalence test (declare that the null is true as a significant result). In the latter case one can show that the power of an equivalence test goes to 1 when the null is in fact true.

Answer 2

I think you’re conflating two different arguments against p-values. Let me spell them out.

• By definition, p is distributed uniformly under the null hypothesis (or as uniformly as possible in discrete settings). So p isn’t going to be a useful measure of evidence in favour of the null when it’s true. Evidence in favour of the null isn’t going to accumulate as we collect data, as p just makes a random walk bounded by 0 and 1. I don’t see where the your source mentions bias in this context.

• There exist mathematical theorems showing that p is much less than the posterior of the null for any choices of priors for parameters in the alternative hypothesis. This is what’s meant by overstating the evidence. For example, p = 0.05 under some assumptions corresponds to a null hypothesis that is at least about 30% plausible.

Let me note that p isn’t necessary at all in frequentist inference, and it’s use isn’t supported by any coherent framework. Indeed, there are really two kinds of frequentism here.

• Fisher, in which p is used and used as a measure of evidence. It is this one that the above arguments attack

• Neyman-Pearson, in which if p is used at all it’s used only to define a rejection region. The connection - or lack thereof - between p and evidence is neither here nor there.

Lastly, let me clarify confusion about the connection between NHST and proof by contradiction. In NHST, when faced with a small p-value, we face the dichotomy

• we observed a small p and the null was true or
• we observed a small p and the null was false

which evidently reduces to nothing more than the fact that we observed small p. It gets us nowhere deductively. Proof by contradiction on the other hand obviously does allow us to deductively prove non-trivial things. What required here isn’t deductive logic - like proof by contradiction - but inductive logic.

Answer 3

For me a core issue here is that the Bayesian criticism of the p-value is based on Bayesian reasoning that a frequentist would not normally accept. For the Bayesian, the "true parameter" is a random variable (as probability distributions are about formalising uncertainty, and there is uncertainty about the value of the true parameter), whereas for the frequentist the "true parameter" is fixed and the basis of probability calculations (as probability distributions are about how data will distribute under idealised infinite replication).

The Bayesians start from a prior distribution over the parameter, which according to frequentist logic does not normally exist (unless we're in a situation where various "true" parameters are indeed generated in some kind of repeatable experiment as in "empirical Bayes" situations).

Updating the prior by the data, the Bayesian will produce a posterior and can then make statements about what the probability is that the true parameter lies in a certain set or takes a certain value. Such statements can not be made in frequentist logic, and surely the p-value doesn't do such a thing.

What's behind the "p-values overstate the evidence" issue is that some Bayesians actually interpret the p-value as (some kind of approximation of) the probability that the null hypothesis is true, in which case they can compare it with the same probability computed by Bayesian logic. Depending on the prior, one could then come to the conclusion that the p-value is too low or too high.

(This paragraph added after comments:) The connection between this and the statement about "evidence" is that some Bayesians tend to think that only probabilities that hypotheses are true (and certain quantities derived from them) qualify as valid measurements of evidence. This means that in order to accept the p-value as a measure of evidence, they need to interpret the p-value in this way. A frequentist can still think of a p-value as a measurement of evidence, but this measurement would then be something essentially different, as probabilities of hypotheses being true don't normally make sense in frequentist logic.

The problem here is that this (a probability of the null hypothesis being true) is not what the p-value is; according to frequentist logic there is no such thing as a "true prior" that could be used as a basis for this, and the p-value is a probability computed assuming the null hypothesis to be true, rather than a probability that the null hypothesis is true. Therefore a frequentist shouldn't accept the Bayesian computation as "what the p-value should be". The Bayesian argument (not shared by all Bayesians!) here is that a Bayesian interpretation of the p-value isn't as good as proper Bayesian analysis, but the frequentists can say that the p-value shouldn't be interpreted in this way in the first place.

The Bayesians have a point though in the sense that the p-value is often misinterpreted as a probability of the null hypothesis being true, so their criticism, although not applicable to a correct understanding of the p-value, applies correctly to what some people make of it. (Furthermore Bayesians can claim that parameters should be treated as random variables rather than as fixed, which is a more philosophical discussion and doesn't concern p-values in particular but the whole of the frequentist logic.)

Answer 4

The discussion here is excellent but at the heart of the matter is that in an attempt to "let the data speak for themselves", i.e., to be objective, the frequentist approach jettisons the desire to obtain measures of evidence in favor of assertions. As explained eloquently in Bernoulli's Fallacy, Clayton takes us step by step through statistical history to explain how this happened and what harm it has done. One of the harms is that outside information was prevented from being brought into the analysis. One of his excellent examples is ESP research where he shows that obtaining a low p-value is meaningless without factoring in the low likelihood that the laws of physics are allowed to be suspended. So one can analyze all day the amount of evidence in a p-value but I don't think that is quite worth the trouble.

Answer 5

The idea that p-values overstate the evidence against the null hypothesis is partly due to a misunderstanding about the nature of p-values (as others have mentioned). A p-value can be regarded as a measure to address the hypothesis testing question: Is the data consistent with the null hypothesis? Bayesian posterior probabilities are Bayesian answers to a different question: Given the data, what is the relative plausibility of each hypothesis? The key word here is relative. It should hardly be surprising then that answers to such different questions can differ, even within the one school. In short, P-values are about consistency with one hypothesis, and not about the relative plausibility of two hypotheses.

Answer 6

The key to appreciating frequentist inference is to frame it similar to a proof by contradiction. It is by design that the p-value follows a uniform distribution under the null, i.e. $20\%$ of the time you get a p-value less than or equal to $0.2$. It answers the question, "If this other hypothesis is true, how often would I get a result like the one I've just witnessed?" It's like playing devil's advocate. A reality check. If the p-value was not uniformly distributed under the null you would not get this reality check. There is no bias nor is there an "independent property" because the population parameters do not randomly change from one repeated experiment to the next according to anyone's belief.

Bayesians have a very different way of thinking, so it is not surprising that these criticisms would be levied. Bayesian "probability" measures the belief of the experimenter, and beliefs are not facts. Any claim that the p-value overstates the evidence compared to a posterior probability is mistaking beliefs for facts.

There are one-to-one analogs on everything between the two paradigms. Bayesians often criticize frequentism for not incorporating outside information into an analysis, but Bayesian updating of a prior into a posterior maps to a frequentist meta-analysis with p-values and confidence intervals [1] [2]. Bayesians often criticise frequentism for not accounting for all uncertainty in a model when making predictions, but Bayesian predictive distributions map to frequentist prediction intervals [3]. However, this one-to-one mapping is not a reason to use an unfalsifiable subjective definition of probability.

Even in non-normal models, central limit theorem aside, it is possible to construct confidence intervals by inverting a hypothesis test or a cumulative distribution function to find a range of plausible values of a parameter.