1
$\begingroup$

Assume we want to test the directional hypothesis that $µ<0$. From a frequentist angle we use a one-tailed $t$-test and imagine we obtain a 1-sided $p$ value of say 0.07, which then would imply that our mean is not significantly smaller than 0 at the 0.05 level (some might still refer to it as "marginally significant").

Now I was reading the paper "Three Insights from a Bayesian Interpretation of the One-Sided P Value" by Marsman & Wagenmakers (2017), which notes that one-sided $p$ values of 1-tailed tests (i.e. of directional tests) can readily be converted to Bayes factors $K$ of a Bayesian test for direction under a symmetric prior around $µ$ based on the formula

$K = (1-p)/p$

In this example, this would give us a Bayes Factor $K$ of $µ$ being < 0 as opposed to it being >= 0 of (1-0.07)/0.07 = 13, which would imply that there would be 13 times more support for $µ$ being < 0 than $µ$ being >= 0. Following Jeffreys (1998) as well as Kass and Raftery (1995), this would be regarded as "strong evidence" for there being a negative effect as opposed to a positive one as this $K > 10$. In fact their cutoff for a strong effect $(K=10)$ would then correspond with a one-sided $p$ value of 1/11 = 0.09. Decisive evidence they consider to correspond to $K > 100$, which would then imply a one-sided $p$ value of 1/101 = 0.001. Am I correct stating this, and does it illustrate the arbitrariness and lack of concordance of such proposed cutoffs in the frequentist and Bayesian literature? That is, in this case implying a nonsignificant result from a frequentist angle, but a strong effect from a Bayesian one (under a noninformative prior, which I would be happy with in the context of my analysis)?

I was a little surprised by the lower bar that Bayesians would put on inferring a strong effect than frequentists, given that I've also regularly seen the argument that $p$ values around 0.05 would correspond to quite modest Bayes factors, and that the $p$ value cutoff for significance should therefore be taken as more stringent than 0.05. For example, in this paper, "Evidence From Marginally Significant t Statistics" (Johnson 2019). Am I correct that this difference in conclusion is due to the fact that Marsman & Wagenmakers (2017) consider directional tests (H0: effect in one direction, Ha: no effect or effect in the opposite direction), whereas Johnson (2019) consider tests for the existence of an effect (H0: there is no effect, Ha: there is an effect)? [This is what I gathered based on my reading of Held & Ott, 2018, "On p-Values and Bayes Factors" at least]

I was also wondering, given that maximum likelihood estimates coincide with the MAP estimate in a Bayesian analysis under a non-informative prior (and that Bayesian 95% credible intervals are numerically very close to 95% MLE confidence intervals derived under a non-informative prior), whether $p$ values deriving from maximum likelihood analyses (and most frequentist tests can be stated in MLE terms) can not always be restated as Bayes factors derived under a non-informative prior? If so, is MLE then not merely a special case of Bayesian analysis, and what's the big deal of pitting those two school of thoughts against each other (aside from their choice of a different prior, which in MLE is taken as non-informative based on the lack of information you have about this, and in Bayesian analysis can be specified in a somewhat more subjective way, taking into account your subjective beliefs)?

$\endgroup$
  • $\begingroup$ "...does it illustrate the arbitrariness of such proposed cutoffs in both frequentist and Bayesian statistics" Well, most (if not all) cutoffs in statistics are arbitrary. But I'd add that plenty of Bayesians don't really like Bayes factors, especially because using them with cutoffs just recreates some of the problems with p-values under a different label. $\endgroup$ – mkt - Reinstate Monica Sep 28 '19 at 19:14
  • $\begingroup$ Regarding likelihoodists vs. Bayesians: my impression is that those two schools don't really have such a big beef with each other. Their disagreement is (obviously) about the prior. Bayesians frequently argue against the use of noninformative priors and so the two approaches do differ meaningfully, but they are far closer to each other than they are to frequentists. $\endgroup$ – mkt - Reinstate Monica Sep 28 '19 at 19:17
  • $\begingroup$ @mkt Thanks for that - that makes sense! But I am correct then that in this example frequentists would conclude that there was no significant effect, whereas Bayesians would conclude there is a strong effect? Regarding the difference between the frequentist school and the maximum likelihood school - is it still typically not the case that all typical frequentist hypothesis tests can also be derived from a MLE angle? E.g. one could fit a gaussian error GLM under a MLE framework as opposed to carrying out a t-test under a frequentist framework? $\endgroup$ – Tom Wenseleers Sep 29 '19 at 8:24
  • $\begingroup$ I hesitate to speak on behalf of all Bayesians, but I would imagine that their conclusion would depend on (1) their prior, and (2) the nature of the problem, since 'strong' is a judgement that should include notions of effect size (and this is true for a principled frequentist analysis as well). So there's insufficient information in your example to decide whether the effect is strong or not, especially if you prefer informative priors and dislike Bayes factors (which I think is a very reasonable position). $\endgroup$ – mkt - Reinstate Monica Sep 29 '19 at 9:01
  • 1
    $\begingroup$ Thanks for the pointers! In my case, I was cool if the assumed prior was uninformative and within the context if my biological problem I would say that the observed effect size was such that it was definitely of interest, but it's all a bit subjective of course and all these cutoffs seem to be purely a matter of convention, and as you say might well be looked at in different fields (in biology we are often already happy explaining 5% of the variation, but I doubt if a particle physicist would be happy with that :-))... $\endgroup$ – Tom Wenseleers Sep 29 '19 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.