I understand that there are multiple comparisons floating around between Fisher's Exact Test and Bayesian A/B Testing, here's an example. While I do understand these are fundamentally making different distributional assumptions about the data, I'm still confused as to which assumptions are driving the difference. What would be illuminating is an explanation about:

  1. Which "data generating story" corresponds to which test and why they're fundamentally different
  2. Which "data generating story" and therefore modeling approach is more appropriate for trial data such as below

Let's focus on a very specific and currently relevant data set: using Hydroxychloroquine as post-exposure prophylaxis against COVID-19. The following data were presented in this paper:

  • Incidence of new illness (COVID-19) for participants receiving hydroxychloroquine (49 of 414 [11.8%])
  • Incidence of new illness (COVID-19) for participants receiving placebo (58 of 407 [14.3%])
  • The absolute difference was −2.4 percentage points (95% confidence interval, −7.0 to 2.2; P=0.35
  • In the "STATISTICAL ANALYSIS" section, "We assessed the incidence of Covid-19 disease by day 14 with Fisher’s exact test"

Fisher's Exact Test

I was able to run a two-sided fisher.test in R and get the advertised P=0.35.

I conceptually understand that the "null hypothesis" being tested is if these two samples (hydroxychloroquine vs. placebo patients getting sick) come from the same distribution. From the wiki article on the test: "The formula above gives the exact hypergeometric probability of observing this particular arrangement of the data, assuming the given marginal totals". Applying this to our specific example:

  • a = patients who got hydroxychloroquine and got sick = 49
  • b = patients who got placebo and got sick = 58
  • c = patients who got hydroxychloroquine and did not get sick = 365
  • d = patients who got placebo and did not get sick = 349

Converting the data above to match the hypergeometric PMF/CDF variables:

  • k = a = 49
  • n = a + c = 414
  • N = a + b + c + d = 821
  • K = a + b = 107

I was able to get the one-sided "less" fisher.test by computing the hypergeometric CDF using this calculator. This ties out with R's ~17.8%.

Conceptually this to me means: "The probability of observing 49 or less people taking hydroxychloroquine and getting sick by pure chance in a sample of 414 from a population of 821 patients in which 107 total people got sick".

The two-sided value reported in the paper is roughly twice this. This is roughly analogous to something like a two sided t-test. I understand this is a bit awkward to do as described in the wiki entry.

Bayesian A/B Testing

Let us ignore the problem of picking priors for now. Using the data above we can define two distributions of the proportion of people who get sick:

  • B_hydroxychloroquine ~ Beta(49 + 1, 365 + 1)
  • B_control ~ Beta(58 + 1, 349 + 1)

We can estimate the probability (B_hydroxychloroquine - B_control) > 0 via simulation. It turns out to be ~15.3% (+/- 0.1% from simulation variance).

Conceptually this to me means: "The probability that the proportion of people taking hydroxychloroquine and getting sick is greater than the proportion of people taking placebo and getting sick".

What would be the equivalent to the two-sided fisher.test for the Bayesian A/B example? (My only guess is just to multiply the probability by two).

Code Sample


n_sims <- 1e5

treat_pts <- 414
treat_affected <- 49

cont_pts <- 407
cont_affected <- 58

treat_dist <- rbeta(
  n = n_sims,
  shape1 = treat_affected + 1,
  shape2 = (treat_pts - treat_affected) + 1

sprintf('%.1f%% treat pts affected', mean(100*treat_dist))

cont_dist <- rbeta(
  n = n_sims,
  shape1 = cont_affected + 1,
  shape2 = (cont_pts - cont_affected) + 1

sprintf('%.1f%% cont pts affected', mean(100*cont_dist))

ratio <- treat_dist/cont_dist

    'Raw treat:cont death diff %.2f, 95%% CI %.2f to %.2f\nProb(treat:cont) > 1 = %.3f\n',
    quantile(ratio, probs = 0.025),
    quantile(ratio, probs = 0.975),
    mean(ratio > 1)


diff <- 100*(treat_dist - cont_dist)

    'Raw treat - cont death diff %.2f, 95%% CI %.2f to %.2f\nProb(treat - cont) > 0 = %.3f\n',
    quantile(diff, probs = 0.025),
    quantile(diff, probs = 0.975),
    mean(diff > 0)


contingency_table <- 
  matrix(c(treat_affected, treat_pts-treat_affected, 
           cont_affected, cont_pts-cont_affected),
         nrow = 2,
         dimnames = list(Status = c("Affected", "Not Affected"),
                         Group = c("Treatment", "Control")))

fisher.test(contingency_table, alternative = 'less')
fisher.test(contingency_table, alternative = 'greater')
fisher.test(contingency_table, alternative = 'two.sided')

1 Answer 1


Which "data generating story" corresponds to which test and why they're fundamentally different?

Which "data generating story" and therefore modeling approach is more appropriate for trial data such as below?

Please, let us first get the idea of a data generating story out of your head. There is no data generation story here, only inferential testing. I will show you why at the end of this post.

Using Keynesian notation, a Bayesian posterior probability is denoted $\Pr(\theta|X)$ where is some data set. Using the same notation, a Frequentist test would be $\Pr(X|\theta)$. The methods are orthogonal to one another.

Both methods use the sample space and the parameter space. However, the Frequentist test uses the sample space and fixes the parameters to the null hypothesis. Only one element of the parameter space is used. The Bayesian test uses the parameter space and only considers the subset of the sample space congruent to the observed data.

By holding the parameter fixed, if you have chosen the true value of the parameter, then you can estimate the probability of seeing data as extreme or more extreme than the data that you saw due to chance alone. You are treating the null value(s) of the parameters as true.

In Bayesian estimation you are holding the sample as fixed. There is nothing random about it, after all, you saw it. It is a fact. Because there is no equivalent to a null hypothesis in Bayesian thinking, you are testing the probability that a parameter is in a subset of the parameter space.

Now let us consider our specific problem from a Frequentist perspective. If we want to know if hydroxychloroquine is effective in preventing the disease, then we want to know if the incidence of disease with it is less than the incidence of the disease under a placebo.

Because the null hypothesis method is the conceptual kin to modus tollens, we want to assume the opposite is true and then have nature reject the null if it is not true.

For notational purposes let $\theta_1$ be the population parameter when hydroxychloroquine is used preventatively. Likewise, let $\theta_2$ be the population parameter when a placebo is used. Note that I did not say that that $\theta_2$ is the population parameter without treatment. What if sugar pills prevent the disease?

So our null, under the Fisher Exact Test, is that $\theta_1\ge\theta_2$. The Fisher Exact Test is an odds test, so our null if we use that is $$\frac{\frac{\theta_1}{1-\theta_1}}{\frac{\theta_2}{1-\theta_2}}$$

As an example of the null, let us assume that $\theta_1=.05$ and $\theta_2$=.04. Then our odds ratio is $$\frac{\frac{.05}{.95}}{\frac{.04}{.96}}=1.26$$ The equivalent odds null hypothesis is that the ratio is greater than or equal to one.

We are not testing exact equality unless we are not using it as a drug. Exact equality is that there is no effect. We do not care if there is no effect. We want to know if there is either no effect or harm. The alternative is whether there is a beneficial effect. Because it is an exact test, we have an exact p-value of $p=.1778.$ The implication is that although the odds ratio is approximately $.808$, we cannot reject the idea that there is no effect or that the effect is harmful.

If our concern is factual, then we should tentatively report that we cannot reject the null that the differences are due to chance, if the null is true. For a scientific consensus, hydroxychloroquine should not be used by anyone except in experimental trials, based on this data.

Now let us turn to a subjective question, should a drug manufacturer continue research trials into hydroxychloroquine. The null hypothesis won't answer that. It just says all clinical use should cease unless more controlled research is performed.

The drug manufacturer is interested in whether this is profitable or not. The question is very different. Even if rejected using null hypothesis methods, this does not imply that the research line should be dropped, in and of itself.

You proposed a uniform prior distribution, but I am not so sure that makes sense. Bayesian methods do not consider one hypothesis to be a null and the other an alternative. To allow for that, I am going to run two prior distributions. First, I will do all uniform prior densities. Second, I am going to change the prior on the hydroxychloroquine case to the triangular prior. In essence, instead of a $\beta(1,1)$ prior I am using a $\beta(1,2)$ prior.

The distributions under the uniform prior are shown here for the two parameters. uniform

The distributions under the assumption of a triangular prior for the drug are shown here. triangle

A visual inspection makes it seem likely that the drug, hydroxychloroquine, should be used prophylactically.

An exact solution is known for this case. The posterior distribution of the differences, $\theta_1-\theta_2$ is solved by the convolution of the distributions. The $$\Pr(\theta_1>\theta_2|a,b,c,d)=\int_0^1\beta(z,a,b)I_\beta(z,c,d)\mathrm{d}z,$$ where $a,b$ are the posterior values for the beta density with hydroxychloroquine and $c,d$ are the posterior values for the beta density with the placebo. The function $I_\beta$ is the incomplete beta function, which is the cumulative density function of the beta distribution.

Using a Riemann approximation, $$\Pr(\theta_1>\theta_2|X)=.1529$$ under the uniform prior and .1498 under the triangular. That would be considered substantial evidence under Bayesian hypothesis thinking for the continued testing of hydroxychloroquine. The odds that hydroxychloroquine reduces the incidence of the disease are, under the triangular prior, $5.67:1.$

These are opposing results. Of course, the Frequentist test is $$\Pr(X|\theta_1-\theta_2\ge{}0)=.1778.$$ The Bayesian test is $$\Pr(\theta_1-\theta_2\ge{}0|X)=.1498$$

Now as to why this is not a data generating story. These are not data generating models. Imagine we expanded our question from conditional incidence of illness to lives saved. For example, for men aged 50, a colonoscopy saves as many lives by catching cancer early as die who were otherwise healthy from complications due to the procedure.

There is nothing in the above model that asks how the data is generated.

The Frequentist result would be to remove it from general use. The Bayesian result would be to continue research. If you look at the implicit Frequentist plots of the binomial distributions under the null and the location of the observations, there is nothing surprising about the failure to reject the null. double_binomial

Neither of these observations would be individually surprising under the null.

So, now, facing contradictory results, which one should you use? It is pretty clear that the Frequentist results could easily be the result of the null being true. It is also pretty clear that the Bayesian results imply the drug works.

The answer is that you should have chosen your method before you looked at the data. The two methods are usually mutually exclusive, from an axiomatic perspective. By knowing the data, you can influence what you choose. That is sort of why asking about the data generating story is a problem. Nature generates data. Probabilities or their statistics do not generate data.

If you have a preferred default action, such as being prejudiced that a drug will kill you or be worthless, then you should be using a Frequentist method. It institutionalizes your prejudice as the null hypothesis. Facing ignorance, you do not put unknown substances in your body.

On the other hand, imagine you are not invested in either result. You want to investigate the case. You are not planning on taking the drug, you are going to give it to foolish test subjects that have signed a waver. You want to form a better personal opinion about the true state of nature. Maybe you believe the drug is at least a little helpful. You want that personal belief involved in your decisions. Facing ignorance, you want to get the best estimator, using as much information as you can get. You want to make the best action after you have collected the data. Maybe you are not totally ignorant, just mostly ignorant.

Facing ignorance, the Frequentist method will tell you if you should reject your default action. You will only appear foolish up to $\alpha$ percent of the time if you perform this research an infinite number of times.

Facing either ignorance or less knowledge than you would like, the Bayesian estimate will give you the best estimate of how to update your beliefs about the true state of the world without showing favoritism to any decision other than from the result of a priori knowledge.

You should be careful in interpreting the results as comparable. They are not. The Frequentist probability is the frequency of times that such a result would happen if the null is true due to chance effects alone. It is a frequency statement. You would likely set your $\alpha$ very low so that you don't kill people unnecessarily, such as $\alpha=.01$. The Frequentist result is not significant.

The Bayesian result would be that there is substantial evidence that the drug worked. In no sense would it be decisive and Bayesians lack a concept of significance, but depending on your cost function, this result would imply testing should continue under a mild cost function. If this drug is cheap, then you should multiply your sample by at least ten and try again.

EDIT I find statements like, "we find some evidence in support of," when, at the same time, you report no significance. While I strongly agree that significance is misused, I don't think additional language beyond reporting the effect size is warranted. With that said, you should also include post hoc power analysis.

Pearson and Neyman's Frequentist method is binary. You are in the acceptance region or you are in the rejection region. If you set $\alpha$ then that is your standard. Not being able to reject the null does not imply that there is no effect. Indeed, Fisher reported a p-value instead because he did not control for type 2 error.

A non-finding is a finding, especially in this case. The Bayesian perspective clouds the matter. The Bayesian answer would be that you should get a larger sample. The Bayesian answer is that it is worth researching, but nothing strong enough to make a definitive statement. You certainly would not recommend someone use the drug based on this result alone.

It also does not answer any contextual questions. Imagine that the drug works, but there is another drug that reduces the rate of infection by 90%. Then, while it works, it is like recommending someone purchase a backboard wagon when a Lamborghini is the alternative.

What the null hypothesis is really doing is saying, "I concede the argument, let us presume the drug is either worthless or dangerous to patients. However, before we stop, let us collect data and see if nature rejects that assumptions, at least in a probabilistic sense of the idea.

To come back after the fact and say, "I conceded, I didn't prove my case, but I still want you to accept my alternative hypothesis might be okay" is a little bit disingenuous unless you brought more data.

  • $\begingroup$ I have to edit this. It contains an error. $\endgroup$ Commented Sep 22, 2020 at 20:04
  • $\begingroup$ I fixed the error. $\endgroup$ Commented Sep 22, 2020 at 20:14
  • $\begingroup$ This is an excellent and detailed response, thank you for taking the time to put this together. I think what was particularly illuminating was phrasing it in terms of P(data | parameters) vs. P(parameters | data). Respectfully, I disagree with your assertion that "this is not a data generating story". The frequentist approach's null states that these data are drawn out of a hypergeometric distribution based on the fixed row/column totals. The Bayesian likelihood asks about plausible probability values assuming the data are binomially distributed. $\endgroup$
    – DataDan01
    Commented Sep 23, 2020 at 3:35
  • $\begingroup$ Could you please clear up the last plot? What exactly is meant by "implicit Frequentist plots"? I thought the Fisher test specifically focused on pooling the row/column totals into a hypergeometric distribution, which is different from the binomial distribution. $\endgroup$
    – DataDan01
    Commented Sep 23, 2020 at 3:40
  • $\begingroup$ @DataDan01 you are correct, the hypergeometric is the appropriate distribution for the test, except there are two things added to consider. First, this is a test of a drug. It isn't an urn. None of the patients could have been drawn with replacement. It is drawn from a population, it isn't the population. Second, the sampling distribution is the hypergeometric for the test. I wrote "implicit" because it is not the sampling distribution implied by the test because I split the data as if the other row did not exist. $\endgroup$ Commented Sep 23, 2020 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.