# Frequentest and Bayesian analyses contradicting each other

I have the following set of data which I'm trying to analyse:

                    | Treatment A | Treatment B |
did the thing       |     284     |     333     |
didn't do the thing |    2554     |    2509     |
n                   |    2838     |    2842     |


Normally I'd use a fisher or chi square test to see if there is a difference between the treatments. In this case a two tests give a p values of 0.04065 and 0.04254 respectively, so I should reject the null hypothesis and say that there is a difference between the two treatments.

Lately though I've been dabbling with the Bayesian approach to analysis. When I calculate a Bayes Factor for this set of data though I get a BF = 0.21, implying a moderate level of evidence for the null hypothesis.

I guessing this difference in inference from the data is down to the differences in philosophy between the frequentest and Bayesian approaches to statistics but don't have enough experience to know how to continue from here as I kinda thought they were really just two different ways of skinning the same cat.

Has anyone had any experience in this situation? Code I used in R below:

# Input Data
line1 <- c(284, 333)
line2 <- c(2554, 2509)
n.line <- c(2838, 2842)

# Contingency Tables
b.dat <- rbind(line1, n.line)
f.dat <- rbind(line1, line2)

# Bayes Factor Analysis
contingencyTableBF(as.matrix(b.dat), priorConcentration = 1, sampleType = "poisson")

# Fisher Test
fisher.test(f.dat)
chisq.test(f.dat)

• I can't say what it might be in your case but when the sample size is small the prior dominates the determination of the posterior distribution and the data seems to be in conflict with the prior. – Michael R. Chernick Mar 26 '18 at 23:59
• This is a nice example of a situation in which a Bayesian might decry the "exaggeration of evidence" by a P-value. The strength of evidence needed for a p-value of 0.04 is pretty weak by other indices, and a Bayes factor of less than 10 (more than 0.1) is to be expected. – Michael Lew Mar 27 '18 at 2:40
• This is called the Jeffrey-Lindley Paradox. – jaradniemi Mar 27 '18 at 20:03

The coding of this package is rather strange as Bayes Factors ignore the prior density of the null and the alternative hypotheses. Nonetheless, what you are seeing happens quite a bit.

If you look at the output of your R code you will note that your interval trivially misses overlapping with 1. It is just barely over the threshold of significance. What you are seeing is the impact the strong effect of the null can have, or conversely the weak effect created when one lacks a null.

If you do not mind using Bayesian notation as this would not be how it would be presented in Frequentist thought, then what Fisher's exact test is asserting is $\Pr(data|\mu=1)$. You are asserting a very strong thing, that $\mu\equiv{1}$. That is information in and of itself. It drives what is acceptable or unacceptable in the test. The Bayesian is testing the weaker model of $\Pr(\mu-\epsilon\le{}1\le\mu+\epsilon|data)$.

Think about what the conflicting statements would imply in English language terms. They would be equivalent if a bit outside the rules of using terms of art to being, "if the null is true then this data is unusual," and "the data does support that $\mu$ is in the region of 1" The former quote is Frequentist and the later Bayesian. The former asserts the strong exact relations, the latter allows it to be nearby but not 1.

My guess is that you should be using the Fisher Exact Test because that seems to be what you are testing and not the Bayesian test. Fisher's Exact Test is a "no effect," test. You are asserting that there is no effect at all. You are not discussing the size of the effect. The result would tend to imply that there is an effect. The Bayesian test would tend to support that either there is no effect or it is a very small effect. Even a very small or trivial effect is an effect. This allows both to be correct at the same time.

You should note that this does not imply that Fisher's method is good for exact solutions and Bayes for inexact solutions, it is just how it plays out in this specific test scenario. It is true, however, that Bayesian methods do not handle sharp null hypotheses as well as non-Bayesian methods such as the one here.

There is also a question as to whether you should have used Bayes, but with a completely different set of hypotheses than the one you used. Imagine treatment A is the pre-existing standard treatment and B is a new treatment. Then you do not want to know if they are the same, you want to know if B is better. In that case, your Bayesian hypothesis would have been different because you want to know if $B\gg{A}$. That would have triggered a very different result for both the Frequentist and the Bayesian hypothesis tests.