# Bayes theorem an experiment design

Let's say I design an experiment to test the hypothesis, "Agent Z causes cancer". Of $N$ people in the experiment group, $x$ get cancer. Of $N$ people in the control group, $y$ get cancer. The results are that $x > y$ and $N \gg x$.

Pretty standard so far. I'm interested in a Bayesian interpretation of the results of this experiment, so I come up with the following statement about the "probability that my hypothesis is correct ($P(H_{\rm true})$) given the evidence ($e$)"

$P(H_{\rm true}|e) = \frac{P(e|H_{\rm true})P(H_{\rm true})}{P(e)}$

I'm specifically interested in the quantity $P(e|H_{\rm true})$. What can be said about that quantity given the information available? Is there a general way to calculate or estimate it, knowing nothing more about the relationship between Agent Z and cancer? If not, what more would you need to calculate or estimate it?

• This being a Q/A site, if you think your question was answered below you could accept one of them, or tell us what is still unclear. – conjugateprior May 20 '14 at 12:58
• I think the reason I hesitated, is that your answer, while certainly more constructive the other answer, it appears to make a lot of big assumptions. I fear that the approach you describe is therefore no better than simply saying "no, in general there is no way to calculate or bound the probability without knowing more about how (as @Bitwise puts it) the phenomenon generates observations." Is that a bad interpretation of your answer? – brooks94 May 20 '14 at 13:08
• Well, there are assumptions I can make about the sampling which will destroy or reverse any possible conclusions. The most relevant of these are rampant selection bias into the treatment group and/or individual effect heterogeneity for those therein. This is, of course, not a statistical issue but a causal identification issue, good discussion of which can be found in the first few chapters of Morgan & Winship's textbook. – conjugateprior May 20 '14 at 13:16
• On the other hand, I inferred what I did about randomisation and control from your question's use of the word 'experiment'. It might be useful to point out that under the conditions I sketched, an average treatment effect of some kind is inferable without knowing the mechanics of how Z actually does its cancerous thing. – conjugateprior May 20 '14 at 13:20

First, any causality claims come from the details of the experimental setup, sampling, confounders, etc. about which Bayes, as statistical inference method has essentially nothing to say.

Nevertheless, here's a sketch: In the simplest setup there are $2N$ randomly chosen people, $N$ of whom are exposed to agent $Z$. Assume there is a baseline probability of getting cancer without $Z$ which is common to everyone involved, and the effect of $Z$ is, possibly, to raise this probability.

Your observations are then Binomial: $x$ cancers for $N$ opportunities in the treatment group and $y$ cancers in $N$ opportunities in the control group. You might then model the probability $p$ of getting cancer by relating the log odds of a getting cancer to an intercept parameter $\alpha$ that represents the baseline chance of getting cancer and a dummy variable with coefficient $\beta$ indicating being in the treatment group. This is a standard logistic regression model and it provides the likelihood.

A simple Bayesian approach to figuring out whether $\beta$ were truly positive i.e. whether $Z$ caused cancer, would be to put priors on $\alpha$ and $\beta$, compute a posterior distribution over the two unknown parameters given the data and model, and then marginalise out $\alpha$. If sufficiently much of the probability mass of the resulting posterior over $\beta$ were greater than zero you might conclude that Z caused cancer.

If you want to think in terms of $H$s, then I suppose $H_{true}$ would be that $\beta > 0$. You'd have to choose some decision rule for how much of the posterior probability would have to end up on this section of the possible $\beta$ values before declaring it true. Actually, you get rather more useful information than that from this analysis, e.g. about the range of probable effect sizes and their direction.

All in all, the resulting analysis looks pretty much like the frequentist one, but that's because I've made all the usual convenient sampling assumptions and glossed over all the actual medical issues that would make it a harder inference problem.

$P(e|H)$ represents a probability distribution used to model how the phenomenon "generates" observations. For example, if Z causes cancer stochastically, how would that work? Would it follow a normal distribution? You would need to make some assumption about the form of this distribution.