15
$\begingroup$

I recently started reading the Bayesian criticism of the p-value and it seems that there is a lot of discussion around the fact that a frequentist approach is not that good when the Null Hypothesis is true.

For instance in this paper the authors write that "p-values overstate the evidence against the null [...] this does not have to do with type-I or type-II errors; it is an “independent” property of p-value."

To illustrate this point, the authors show that when the null is true, the p-value has a uniform distribution.

What I do not get is that even when the null is true, a frequentist approach, thanks to the Central Limit Theorem, is still able to construct confidence intervals that includes 0 (non-significance) at the appropriate $\alpha$ level.

I do not get why the fact that the p-value is uniform when the null is true shows that a frequentist approach is biased. And what does it mean "independent property of p-value"?

enter image description here

library(tidyverse)
library(broom)

n=1000
x = rnorm(n,100,30)
d = 0
y = x*d + rnorm(n,0,20)
df = data.frame(y,x)
plot(x,y)
abline(lm(y~x), col = 'red')

r = replicate(1000, sample_n(df, size = 50), simplify = F)
m = r %>% map(~ lm(y~x,data = .)) %>% map(tidy)

# Central Limit Theorem
bind_rows(.id = 'sample', m) %>% filter(term =='x') %>% ggplot(aes(estimate)) + facet_grid(~term) + geom_histogram()

s = bind_rows(.id = 'sample', m) %>% filter(term =='x')
s$false_positive = ifelse(s$p.value < 0.05, 1, 0)
prop.table(table(s$false_positive))

# uniform
hist(s$p.value, breaks = 50)
$\endgroup$
7
  • 6
    $\begingroup$ Deborah Mayo (and discussants) on this issue: errorstatistics.com/2017/01/19/… Mayo thinks that the idea that p-values "overstate the evidence against the null" is a fallacy, and I tend to agree. $\endgroup$ Aug 4 at 23:33
  • $\begingroup$ In the errorstatistics article they are complaining about the realism of the .5 prior on the perinull, that maybe it should be lower. But in many applications, such as genomic screening, where a prior most coding regions are unrelated to the phenonmon of interest, the perinull prior is actually much higher than .5. And then there is ESP studies ... $\endgroup$ Aug 5 at 0:49
  • 10
    $\begingroup$ "authors show that when the null is true, the p-value has a uniform distribution." This is not some suprising feature discovered by the authors. It is by design (it's also not universally the case, but when it's true it's because it is meant to do just that). This would be as strange as saying someone who is opposed to cars has shown that "brakes serve to impede the forward progress of the car" (Yes, that's exactly what they're supposed to do, this is not news). $\endgroup$
    – Glen_b
    Aug 5 at 3:52
  • 3
    $\begingroup$ In order for the p-value to be uniform under $H_0$ the test statistic must be continuous and the test exact. $\endgroup$
    – BruceET
    Aug 5 at 5:57
  • 3
    $\begingroup$ Let me contribute a recent paper by Sander Greenland in which he addresses some common criticism of $p$-values (including that they overstate the evidence). $\endgroup$ Aug 5 at 8:42
19
$\begingroup$

The point that the authors are trying to make is a subtle one: they see it as a failure of NHST that, as $n$ gets arbitrarily large, the $p$-value doesn't tend to 1. It's a bit surprising that this doesn't contain any discussion of equivalence testing. To me it's somewhat obvious and reasonable that the p-value maintains its uniform distribution when the null is true considering larger and larger $n$. Large $n$ means having sensitivity to detect smaller and smaller effects, while the false positive error rate remains fixed. So under the somewhat constrained setting of the null being exactly true, the behavior of the $p$-value distribution doesn't depend on $n$ at all.

  1. NHST is, in my mind, desirable specifically because there's no way of declaring a null hypothesis to be true, as my experimental design is setup specifically to disprove it. A non-significant result may mean that my experiment was underpowered or the assumptions were wrong, so there are risks associated with accepting the null that I'd rather not incur.

  2. We never actually believe that the null hypothesis is true. Typically failed designs arise because the truth is too close to the null to be detectable. Having too much data is kind of a bad thing in this case, rather there's a subtle art in designing a study to obtain only enough sample size so as to reject the null when a meaningful difference is present.

  3. One can design a frequentist test that sequentially tests for differences (one or two tailed), and depending on a negative result, performs an equivalence test (declare that the null is true as a significant result). In the latter case one can show that the power of an equivalence test goes to 1 when the null is in fact true.

$\endgroup$
11
  • 2
    $\begingroup$ I suggest replacing normal -> usual to avoid above confusion $\endgroup$
    – innisfree
    Aug 5 at 8:51
  • $\begingroup$ I’m not sure this really addresses the criticism - is p a useful measure of evidence? The criticisms say that it isn’t. $\endgroup$
    – innisfree
    Aug 5 at 10:19
  • 2
    $\begingroup$ I struggle with the point that we never believe the null to be true. I am a social scientist and in many cases you seriously test the null to be true, for instance when you study if racial or sex differences in earnings have converged over time. $\endgroup$
    – giac
    Aug 5 at 10:52
  • $\begingroup$ "Large n means having sensitivity to detect smaller and smaller effects, while the false positive error rate remains fixed", how can that be true!? it is so un-intuitive! What about the confidence intervals getting smaller as $n$ grows? This surely must reduce the false positive rate?? $\endgroup$
    – giac
    Aug 5 at 10:55
  • 1
    $\begingroup$ @AdamO it seems from your comments (which is expected) that there is a huge gap between statistical theory and applied quantitative social science. First, I take your point that any hypothesis can be disproven, but even the top economists never lay as precise numbers are your 20%, (racial, sex, class, ...) gaps are almost never specified in detail. Often you see "statistical" differences of very small effect size, which leads to your second point. Rarely in practice the critical value is adapted to $n$, it's just 0.05 by default, which might be problematic, but that's what is done in practice $\endgroup$
    – giac
    Aug 5 at 15:59
8
$\begingroup$

I think you’re conflating two different arguments against p-values. Let me spell them out.

  • By definition, p is distributed uniformly under the null hypothesis (or as uniformly as possible in discrete settings). So p isn’t going to be a useful measure of evidence in favour of the null when it’s true. Evidence in favour of the null isn’t going to accumulate as we collect data, as p just makes a random walk bounded by 0 and 1. I don’t see where the your source mentions bias in this context.

  • There exist mathematical theorems showing that p is much less than the posterior of the null for any choices of priors for parameters in the alternative hypothesis. This is what’s meant by overstating the evidence. For example, p = 0.05 under some assumptions corresponds to a null hypothesis that is at least about 30% plausible.

Let me note that p isn’t necessary at all in frequentist inference, and it’s use isn’t supported by any coherent framework. Indeed, there are really two kinds of frequentism here.

  • Fisher, in which p is used and used as a measure of evidence. It is this one that the above arguments attack

  • Neyman-Pearson, in which if p is used at all it’s used only to define a rejection region. The connection - or lack thereof - between p and evidence is neither here nor there.

Lastly, let me clarify confusion about the connection between NHST and proof by contradiction. In NHST, when faced with a small p-value, we face the dichotomy

  • we observed a small p and the null was true or
  • we observed a small p and the null was false

which evidently reduces to nothing more than the fact that we observed small p. It gets us nowhere deductively. Proof by contradiction on the other hand obviously does allow us to deductively prove non-trivial things. What required here isn’t deductive logic - like proof by contradiction - but inductive logic.

$\endgroup$
9
  • $\begingroup$ "So [the p-value] isn’t going to be a useful measure of evidence in favor of the null when it’s true," except that if the null is true we would see a p-value less than or equal to that observed 100(p)% of the time. The p-value shows the plausibility of the hypothesis given the data. See confidence curves. $\endgroup$ Aug 5 at 22:37
  • 1
    $\begingroup$ The p value doesn’t show the plausibility of the null given the data by any non-trivial definitions or meanings of plausibility, or of given that I’ve ever heard. $\endgroup$
    – innisfree
    Aug 6 at 5:48
  • $\begingroup$ So if I test one hypothesis and it has a p-value of 0.40, I use the same experimental result to test another hypothesis and it has a p-value of 0.04, and a third hypothesis has a p-value of 0.0000000000000001, all of these hypotheses do an equally good job explaining the observed data? All of these hypotheses are equally plausible? See Fisherian frequentism and confidence curves. You even listed Fisher in your response. $\endgroup$ Aug 6 at 13:36
  • $\begingroup$ @GeoffreyJohnson There is also a question of how likely the hypothesis is to start with. This is part of the Bayesian criticism of the p-value. $\endgroup$
    – Andrea
    Aug 6 at 16:35
  • $\begingroup$ The hypothesis is either true or it is false, unless the parameter is an unrealized random variable that exists in a state of superposition. See quantum mechanics. $\endgroup$ Aug 6 at 16:38
4
$\begingroup$

The discussion here is excellent but at the heart of the matter is that in an attempt to "let the data speak for themselves", i.e., to be objective, the frequentist approach jettisons the desire to obtain measures of evidence in favor of assertions. As explained eloquently in Bernoulli's Fallacy, Clayton takes us step by step through statistical history to explain how this happened and what harm it has done. One of the harms is that outside information was prevented from being brought into the analysis. One of his excellent examples is ESP research where he shows that obtaining a low p-value is meaningless without factoring in the low likelihood that the laws of physics are allowed to be suspended. So one can analyze all day the amount of evidence in a p-value but I don't think that is quite worth the trouble.

$\endgroup$
7
  • $\begingroup$ What do you mean by "outside information"? $\endgroup$
    – giac
    Aug 5 at 12:51
  • $\begingroup$ I presume he means prior information that we might have about the plausibility of the null and alternative $\endgroup$
    – innisfree
    Aug 5 at 13:00
  • $\begingroup$ Using a frequentist approach prevents "irrelevant outside information" from being brought into the analysis. There is nothing preventing a meta-analysis of relevant information. $\endgroup$ Aug 5 at 13:41
  • $\begingroup$ I think there is. Meta analysis is a special case of outside information that usually rests on an exchangeability assumption and uses data from other studies. I'm speaking of more general cases than that, including outside information where there are no data, i.e., no previous studies. $\endgroup$ Aug 5 at 14:40
  • 2
    $\begingroup$ The apparent inability to incorporate or respond to "outside information" lies ENTIRELY with the Neyman-Pearson hypothesis test. Fisherian (or should I say neo-Fisherian?) p-values are simply indices of evidence (against the null according to the data and model) and that index of evidence can be (should be) used in conjunction with all other sources of relevant information for a scientific inference. Do not conflate p-values with an automatic decision rule hypothesis test. See this open access chapter for a full explanation link.springer.com/chapter/10.1007/164_2019_286 $\endgroup$ Aug 5 at 21:58
4
$\begingroup$

For me a core issue here is that the Bayesian criticism of the p-value is based on Bayesian reasoning that a frequentist would not normally accept. For the Bayesian, the "true parameter" is a random variable (as probability distributions are about formalising uncertainty, and there is uncertainty about the value of the true parameter), whereas for the frequentist the "true parameter" is fixed and the basis of probability calculations (as probability distributions are about how data will distribute under idealised infinite replication).

The Bayesians start from a prior distribution over the parameter, which according to frequentist logic does not normally exist (unless we're in a situation where various "true" parameters are indeed generated in some kind of repeatable experiment as in "empirical Bayes" situations).

Updating the prior by the data, the Bayesian will produce a posterior and can then make statements about what the probability is that the true parameter lies in a certain set or takes a certain value. Such statements can not be made in frequentist logic, and surely the p-value doesn't do such a thing.

What's behind the "p-values overstate the evidence" issue is that some Bayesians actually interpret the p-value as (some kind of approximation of) the probability that the null hypothesis is true, in which case they can compare it with the same probability computed by Bayesian logic. Depending on the prior, one could then come to the conclusion that the p-value is too low or too high.

(This paragraph added after comments:) The connection between this and the statement about "evidence" is that some Bayesians tend to think that only probabilities that hypotheses are true (and certain quantities derived from them) qualify as valid measurements of evidence. This means that in order to accept the p-value as a measure of evidence, they need to interpret the p-value in this way. A frequentist can still think of a p-value as a measurement of evidence, but this measurement would then be something essentially different, as probabilities of hypotheses being true don't normally make sense in frequentist logic.

The problem here is that this (a probability of the null hypothesis being true) is not what the p-value is; according to frequentist logic there is no such thing as a "true prior" that could be used as a basis for this, and the p-value is a probability computed assuming the null hypothesis to be true, rather than a probability that the null hypothesis is true. Therefore a frequentist shouldn't accept the Bayesian computation as "what the p-value should be". The Bayesian argument (not shared by all Bayesians!) here is that a Bayesian interpretation of the p-value isn't as good as proper Bayesian analysis, but the frequentists can say that the p-value shouldn't be interpreted in this way in the first place.

The Bayesians have a point though in the sense that the p-value is often misinterpreted as a probability of the null hypothesis being true, so their criticism, although not applicable to a correct understanding of the p-value, applies correctly to what some people make of it. (Furthermore Bayesians can claim that parameters should be treated as random variables rather than as fixed, which is a more philosophical discussion and doesn't concern p-values in particular but the whole of the frequentist logic.)

$\endgroup$
11
  • 2
    $\begingroup$ Many, many frequentists and practitioners interpret p as a measure of evidence, including Fisher. It’s not just some Bayesians $\endgroup$
    – innisfree
    Aug 5 at 10:46
  • $\begingroup$ @innisfree I am not arguing against interpreting p as a measure of evidence. The issue is that Bayesians tend to think that only a "probability of a hypothesis" qualifies as "measure of evidence" (as far as they think so, I guess not all of them do). For a frequentist, something different can be a measure of evidence. $\endgroup$ Aug 5 at 10:48
  • $\begingroup$ I also struggle with the point that frequentist have no prior. My field is social science, and it seems that in most studies, people test specific prior. It seems to me that a lot of prior are other "groups of reference". For instance, to study the gender pay gap, you take the men's earnings distribution as a prior and compare the women's earning distribution. How is that not a prior? $\endgroup$
    – giac
    Aug 5 at 10:49
  • 2
    $\begingroup$ @giac It is not a prior in the Bayesian sense of being a distribution over possible true parameters. The men's earnings distribution is a distribution of (potential) observations, not of true parameters. $\endgroup$ Aug 5 at 10:51
  • $\begingroup$ I am not sure I get it... I am wondering now if in social science we actually are not interested in "true" parameters but rather in comparing group distributions, whatever the true parameters are. $\endgroup$
    – giac
    Aug 5 at 11:15
0
$\begingroup$

The key to appreciating frequentist inference is to frame it similar to a proof by contradiction. It is by design that the p-value follows a uniform distribution under the null, i.e. $20\%$ of the time you get a p-value less than or equal to $0.2$. It answers the question, "If this other hypothesis is true, how often would I get a result like the one I've just witnessed?" It's like playing devil's advocate. A reality check. If the p-value was not uniformly distributed under the null you would not get this reality check. There is no bias nor is there an "independent property" because the population parameters do not randomly change from one repeated experiment to the next according to anyone's belief.

Bayesians have a very different way of thinking, so it is not surprising that these criticisms would be levied. Bayesian "probability" measures the belief of the experimenter, and beliefs are not facts. Any claim that the p-value overstates the evidence compared to a posterior probability is mistaking beliefs for facts.

There are one-to-one analogs on everything between the two paradigms. Bayesians often criticize frequentism for not incorporating outside information into an analysis, but Bayesian updating of a prior into a posterior maps to a frequentist meta-analysis with p-values and confidence intervals [1] [2]. Bayesians often criticise frequentism for not accounting for all uncertainty in a model when making predictions, but Bayesian predictive distributions map to frequentist prediction intervals [3]. However, this one-to-one mapping is not a reason to use an unfalsifiable subjective definition of probability.

Even in non-normal models, central limit theorem aside, it is possible to construct confidence intervals by inverting a hypothesis test or a cumulative distribution function to find a range of plausible values of a parameter.

$\endgroup$
12
  • 2
    $\begingroup$ There’s so much to unpack here it’s hard to know where to start. I’ll address the similarity with a proof by contradiction in my answer. $\endgroup$
    – innisfree
    Aug 5 at 4:33
  • 2
    $\begingroup$ There is no contradiction or inconsistency between modeling uncertainty about parameters using random variables and believing that those parameters don't change from one experiment to the next. $\endgroup$
    – fblundun
    Aug 5 at 15:51
  • 3
    $\begingroup$ Hi @GeoffreyJohnson. I notice that you've spent the morning editing links to your other answers into each of your answers and questions. There's no need to do this -- people can find the answers you've written by looking at your profile (stats.stackexchange.com/users/307000/geoffrey-johnson). Additionally, I see you're also linking to your published works at the end, but without any apparent connection to how they directly answer the question. This seems a lot like self-promotion and not about answering the question. $\endgroup$
    – Sycorax
    Aug 7 at 18:01
  • 3
    $\begingroup$ This is strictly a Q&A site. Content that does not directly answer the question is subject to removal. The links might be of interest, but do they answer the question? How? This website has search and tagging features which can turn up these and many other relevant pages. In particular, linking to your published works because they are articles about Bayesian statistics, with no content in the answer relating the content of the papers to the question, seems over the line. More information: meta.stackexchange.com/questions/57497/… $\endgroup$
    – Sycorax
    Aug 7 at 18:18
  • 1
    $\begingroup$ Hi @Sycorax. The three publications compare Bayesian and frequentist methods for inference on a parameter, construct ancillary pivotal quantities for prediction, and use meta-analysis for combining information and transfering inference. I pulled directly from these papers when providing my answers. In my opinion they are not simply tangentally relevant to the questions. Do I need to explain this above each link? Are you suggesting I remove the links instead? Thanks for you help. $\endgroup$ Aug 7 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.