Which "data generating story" corresponds to which test and why
they're fundamentally different?
Which "data generating story" and therefore modeling approach is more appropriate for trial data such as below?
Please, let us first get the idea of a data generating story out of your head. There is no data generation story here, only inferential testing. I will show you why at the end of this post.
Using Keynesian notation, a Bayesian posterior probability is denoted $\Pr(\theta|X)$ where is some data set. Using the same notation, a Frequentist test would be $\Pr(X|\theta)$. The methods are orthogonal to one another.
Both methods use the sample space and the parameter space. However, the Frequentist test uses the sample space and fixes the parameters to the null hypothesis. Only one element of the parameter space is used. The Bayesian test uses the parameter space and only considers the subset of the sample space congruent to the observed data.
By holding the parameter fixed, if you have chosen the true value of the parameter, then you can estimate the probability of seeing data as extreme or more extreme than the data that you saw due to chance alone. You are treating the null value(s) of the parameters as true.
In Bayesian estimation you are holding the sample as fixed. There is nothing random about it, after all, you saw it. It is a fact. Because there is no equivalent to a null hypothesis in Bayesian thinking, you are testing the probability that a parameter is in a subset of the parameter space.
Now let us consider our specific problem from a Frequentist perspective. If we want to know if hydroxychloroquine is effective in preventing the disease, then we want to know if the incidence of disease with it is less than the incidence of the disease under a placebo.
Because the null hypothesis method is the conceptual kin to modus tollens, we want to assume the opposite is true and then have nature reject the null if it is not true.
For notational purposes let $\theta_1$ be the population parameter when hydroxychloroquine is used preventatively. Likewise, let $\theta_2$ be the population parameter when a placebo is used. Note that I did not say that that $\theta_2$ is the population parameter without treatment. What if sugar pills prevent the disease?
So our null, under the Fisher Exact Test, is that $\theta_1\ge\theta_2$. The Fisher Exact Test is an odds test, so our null if we use that is $$\frac{\frac{\theta_1}{1-\theta_1}}{\frac{\theta_2}{1-\theta_2}}$$
As an example of the null, let us assume that $\theta_1=.05$ and $\theta_2$=.04. Then our odds ratio is $$\frac{\frac{.05}{.95}}{\frac{.04}{.96}}=1.26$$ The equivalent odds null hypothesis is that the ratio is greater than or equal to one.
We are not testing exact equality unless we are not using it as a drug. Exact equality is that there is no effect. We do not care if there is no effect. We want to know if there is either no effect or harm. The alternative is whether there is a beneficial effect. Because it is an exact test, we have an exact p-value of $p=.1778.$ The implication is that although the odds ratio is approximately $.808$, we cannot reject the idea that there is no effect or that the effect is harmful.
If our concern is factual, then we should tentatively report that we cannot reject the null that the differences are due to chance, if the null is true. For a scientific consensus, hydroxychloroquine should not be used by anyone except in experimental trials, based on this data.
Now let us turn to a subjective question, should a drug manufacturer continue research trials into hydroxychloroquine. The null hypothesis won't answer that. It just says all clinical use should cease unless more controlled research is performed.
The drug manufacturer is interested in whether this is profitable or not. The question is very different. Even if rejected using null hypothesis methods, this does not imply that the research line should be dropped, in and of itself.
You proposed a uniform prior distribution, but I am not so sure that makes sense. Bayesian methods do not consider one hypothesis to be a null and the other an alternative. To allow for that, I am going to run two prior distributions. First, I will do all uniform prior densities. Second, I am going to change the prior on the hydroxychloroquine case to the triangular prior. In essence, instead of a $\beta(1,1)$ prior I am using a $\beta(1,2)$ prior.
The distributions under the uniform prior are shown here for the two parameters.
The distributions under the assumption of a triangular prior for the drug are shown here.
A visual inspection makes it seem likely that the drug, hydroxychloroquine, should be used prophylactically.
An exact solution is known for this case. The posterior distribution of the differences, $\theta_1-\theta_2$ is solved by the convolution of the distributions. The $$\Pr(\theta_1>\theta_2|a,b,c,d)=\int_0^1\beta(z,a,b)I_\beta(z,c,d)\mathrm{d}z,$$ where $a,b$ are the posterior values for the beta density with hydroxychloroquine and $c,d$ are the posterior values for the beta density with the placebo. The function $I_\beta$ is the incomplete beta function, which is the cumulative density function of the beta distribution.
Using a Riemann approximation, $$\Pr(\theta_1>\theta_2|X)=.1529$$ under the uniform prior and .1498 under the triangular. That would be considered substantial evidence under Bayesian hypothesis thinking for the continued testing of hydroxychloroquine. The odds that hydroxychloroquine reduces the incidence of the disease are, under the triangular prior, $5.67:1.$
These are opposing results. Of course, the Frequentist test is $$\Pr(X|\theta_1-\theta_2\ge{}0)=.1778.$$ The Bayesian test is $$\Pr(\theta_1-\theta_2\ge{}0|X)=.1498$$
Now as to why this is not a data generating story. These are not data generating models. Imagine we expanded our question from conditional incidence of illness to lives saved. For example, for men aged 50, a colonoscopy saves as many lives by catching cancer early as die who were otherwise healthy from complications due to the procedure.
There is nothing in the above model that asks how the data is generated.
The Frequentist result would be to remove it from general use. The Bayesian result would be to continue research. If you look at the implicit Frequentist plots of the binomial distributions under the null and the location of the observations, there is nothing surprising about the failure to reject the null.
Neither of these observations would be individually surprising under the null.
So, now, facing contradictory results, which one should you use? It is pretty clear that the Frequentist results could easily be the result of the null being true. It is also pretty clear that the Bayesian results imply the drug works.
The answer is that you should have chosen your method before you looked at the data. The two methods are usually mutually exclusive, from an axiomatic perspective. By knowing the data, you can influence what you choose. That is sort of why asking about the data generating story is a problem. Nature generates data. Probabilities or their statistics do not generate data.
If you have a preferred default action, such as being prejudiced that a drug will kill you or be worthless, then you should be using a Frequentist method. It institutionalizes your prejudice as the null hypothesis. Facing ignorance, you do not put unknown substances in your body.
On the other hand, imagine you are not invested in either result. You want to investigate the case. You are not planning on taking the drug, you are going to give it to foolish test subjects that have signed a waver. You want to form a better personal opinion about the true state of nature. Maybe you believe the drug is at least a little helpful. You want that personal belief involved in your decisions. Facing ignorance, you want to get the best estimator, using as much information as you can get. You want to make the best action after you have collected the data. Maybe you are not totally ignorant, just mostly ignorant.
Facing ignorance, the Frequentist method will tell you if you should reject your default action. You will only appear foolish up to $\alpha$ percent of the time if you perform this research an infinite number of times.
Facing either ignorance or less knowledge than you would like, the Bayesian estimate will give you the best estimate of how to update your beliefs about the true state of the world without showing favoritism to any decision other than from the result of a priori knowledge.
You should be careful in interpreting the results as comparable. They are not. The Frequentist probability is the frequency of times that such a result would happen if the null is true due to chance effects alone. It is a frequency statement. You would likely set your $\alpha$ very low so that you don't kill people unnecessarily, such as $\alpha=.01$. The Frequentist result is not significant.
The Bayesian result would be that there is substantial evidence that the drug worked. In no sense would it be decisive and Bayesians lack a concept of significance, but depending on your cost function, this result would imply testing should continue under a mild cost function. If this drug is cheap, then you should multiply your sample by at least ten and try again.
EDIT
I find statements like, "we find some evidence in support of," when, at the same time, you report no significance. While I strongly agree that significance is misused, I don't think additional language beyond reporting the effect size is warranted. With that said, you should also include post hoc power analysis.
Pearson and Neyman's Frequentist method is binary. You are in the acceptance region or you are in the rejection region. If you set $\alpha$ then that is your standard. Not being able to reject the null does not imply that there is no effect. Indeed, Fisher reported a p-value instead because he did not control for type 2 error.
A non-finding is a finding, especially in this case. The Bayesian perspective clouds the matter. The Bayesian answer would be that you should get a larger sample. The Bayesian answer is that it is worth researching, but nothing strong enough to make a definitive statement. You certainly would not recommend someone use the drug based on this result alone.
It also does not answer any contextual questions. Imagine that the drug works, but there is another drug that reduces the rate of infection by 90%. Then, while it works, it is like recommending someone purchase a backboard wagon when a Lamborghini is the alternative.
What the null hypothesis is really doing is saying, "I concede the argument, let us presume the drug is either worthless or dangerous to patients. However, before we stop, let us collect data and see if nature rejects that assumptions, at least in a probabilistic sense of the idea.
To come back after the fact and say, "I conceded, I didn't prove my case, but I still want you to accept my alternative hypothesis might be okay" is a little bit disingenuous unless you brought more data.
