# Expressing one-sided p values of directional hypothesis tests as Bayes factors

Assume we want to test the directional hypothesis that $$µ<0$$. From a frequentist angle we use a one-tailed $$t$$-test and imagine we obtain a 1-sided $$p$$ value of say 0.07, which then would imply that our mean is not significantly smaller than 0 at the 0.05 level (some might still refer to it as "marginally significant").

Now I was reading the paper "Three Insights from a Bayesian Interpretation of the One-Sided P Value" by Marsman & Wagenmakers (2017), which notes that one-sided $$p$$ values of 1-tailed tests (i.e. of directional tests) can readily be converted to Bayes factors $$K$$ of a Bayesian test for direction under a symmetric prior around $$µ$$ based on the formula

$$K = (1-p)/p$$

In this example, this would give us a Bayes Factor $$K$$ of $$µ$$ being < 0 as opposed to it being >= 0 of (1-0.07)/0.07 = 13, which would imply that there would be 13 times more support for $$µ$$ being < 0 than $$µ$$ being >= 0. Following Jeffreys (1998) as well as Kass and Raftery (1995), this would be regarded as "strong evidence" for there being a negative effect as opposed to a positive one as this $$K > 10$$. In fact their cutoff for a strong effect $$(K=10)$$ would then correspond with a one-sided $$p$$ value of 1/11 = 0.09. Decisive evidence they consider to correspond to $$K > 100$$, which would then imply a one-sided $$p$$ value of 1/101 = 0.001. Am I correct stating this, and does it illustrate the arbitrariness and lack of concordance of such proposed cutoffs in the frequentist and Bayesian literature? That is, in this case implying a nonsignificant result from a frequentist angle, but a strong effect from a Bayesian one (under a noninformative prior, which I would be happy with in the context of my analysis)?

I was a little surprised by the lower bar that Bayesians would put on inferring a strong effect than frequentists, given that I've also regularly seen the argument that $$p$$ values around 0.05 would correspond to quite modest Bayes factors, and that the $$p$$ value cutoff for significance should therefore be taken as more stringent than 0.05. For example, in this paper, "Evidence From Marginally Significant t Statistics" (Johnson 2019). Am I correct that this difference in conclusion is due to the fact that Marsman & Wagenmakers (2017) consider directional tests (H0: effect in one direction, Ha: no effect or effect in the opposite direction), whereas Johnson (2019) consider tests for the existence of an effect (H0: there is no effect, Ha: there is an effect)? [This is what I gathered based on my reading of Held & Ott, 2018, "On p-Values and Bayes Factors" at least]

I was also wondering, given that maximum likelihood estimates coincide with the MAP estimate in a Bayesian analysis under a non-informative prior (and that Bayesian 95% credible intervals are numerically very close to 95% MLE confidence intervals derived under a non-informative prior), whether $$p$$ values deriving from maximum likelihood analyses (and most frequentist tests can be stated in MLE terms) can not always be restated as Bayes factors derived under a non-informative prior? If so, is MLE then not merely a special case of Bayesian analysis, and what's the big deal of pitting those two school of thoughts against each other (aside from their choice of a different prior, which in MLE is taken as non-informative based on the lack of information you have about this, and in Bayesian analysis can be specified in a somewhat more subjective way, taking into account your subjective beliefs)?

• "...does it illustrate the arbitrariness of such proposed cutoffs in both frequentist and Bayesian statistics" Well, most (if not all) cutoffs in statistics are arbitrary. But I'd add that plenty of Bayesians don't really like Bayes factors, especially because using them with cutoffs just recreates some of the problems with p-values under a different label.
– mkt
Sep 28, 2019 at 19:14
• Regarding likelihoodists vs. Bayesians: my impression is that those two schools don't really have such a big beef with each other. Their disagreement is (obviously) about the prior. Bayesians frequently argue against the use of noninformative priors and so the two approaches do differ meaningfully, but they are far closer to each other than they are to frequentists.
– mkt
Sep 28, 2019 at 19:17
• @mkt Thanks for that - that makes sense! But I am correct then that in this example frequentists would conclude that there was no significant effect, whereas Bayesians would conclude there is a strong effect? Regarding the difference between the frequentist school and the maximum likelihood school - is it still typically not the case that all typical frequentist hypothesis tests can also be derived from a MLE angle? E.g. one could fit a gaussian error GLM under a MLE framework as opposed to carrying out a t-test under a frequentist framework? Sep 29, 2019 at 8:24
• I hesitate to speak on behalf of all Bayesians, but I would imagine that their conclusion would depend on (1) their prior, and (2) the nature of the problem, since 'strong' is a judgement that should include notions of effect size (and this is true for a principled frequentist analysis as well). So there's insufficient information in your example to decide whether the effect is strong or not, especially if you prefer informative priors and dislike Bayes factors (which I think is a very reasonable position).
– mkt
Sep 29, 2019 at 9:01
• Thanks for the pointers! In my case, I was cool if the assumed prior was uninformative and within the context if my biological problem I would say that the observed effect size was such that it was definitely of interest, but it's all a bit subjective of course and all these cutoffs seem to be purely a matter of convention, and as you say might well be looked at in different fields (in biology we are often already happy explaining 5% of the variation, but I doubt if a particle physicist would be happy with that :-))... Sep 29, 2019 at 10:33

If so, is MLE then not merely a special case of Bayesian analysis, and what's the big deal of pitting those two school of thoughts against each other (aside from their choice of a different prior,

There is a difference but it is only not appearing when we consider the location parameter. That is an exception and not a general rule or principle that makes confidence intervals and credible intervals the same. In the case of a location parameter then the likelihood function (which equals the posterior for flat prior) $$\frac{\partial F(x|\theta)}{\partial x}$$ and the fiducial distribution $$\frac{\partial F(x|\theta)}{\partial \theta}$$ coincide.

If you consider other parameters then there is a difference between confidence/fiducial intervals and credible intervals, also when you consider a uniform prior.

The reason for the difference is that the fiducial distribution (and related confidence intervals) and posterior distribution (and related credible intervals) relate to different type of probabilities. The confidence intervals condition on the true parameter value, the credible intervals condition on the observation.

In this example, this would give us a Bayes Factor K of µ being < 0 as opposed to it being >= 0 of (1-0.07)/0.07 = 13

I guess that part of the difference here is not so much the use of a Bayesian approach versus a frequentist approach. After all, the credible intervals and confidence intervals coincide here.

I see two other issues.

• The use of Bayes factor for a composite hypothesis. Those K values and characterizations for strength of evidence are different when you consider densities or when you compare cumulative probabilities. Imagine a posterior distribution that is uniform between -0.05 to 0.95 that makes a K value of 19 for hypothesis $$\theta > 0$$ versus $$\theta \leq 0$$. However if we compare two point hypotheses in those ranges, e.g say -0.025 and +0.025 then the K value is 1.
• The p-value and choice of low significance cutoff value is similar to assigning a higher prior to the null hypothesis. If the Bayesian technique would use a prior with more density close to zero, then the Bayes factor would be smaller as well. The discrepancy between the p-value 0.07 that is not deemed significant and the Bayes factor of 13 that is considered strong evidence is in the prior information. If prior information places little weight on either hypothesis, then not so much evidence from data is necessary to get a strong factor.