Hypothesis testing via separate inference for each group and then combining

Suppose there are two groups, A and B, and we are interested in inferring a certain parameter for each one and also the difference between the two parameters. Here we can take a Bayesian perspective and strive for a posterior distribution in each case. I am wondering if the following is a sound way of doing this:

1. estimate the posterior for group A,
2. estimate the posterior for group B, and
3. estimate the posterior of the difference by sampling extensively the first two posteriors and taking the difference.

I am specifically unsure about this kind of divide-and-conquer approach where each group is treated separately, and then the results are combined. Usually, it is done in one take where, perhaps, a linear model is fitted with an indicator for the group membership.

Let me give a simple example. Say, the outcome is binary. One can then use a Bernoulli–beta model to infer the posterior of the success probability, which will be a beta distribution for each group. As the last step, one can sample the two betas and get a posterior for the difference.

3 Answers

The approach you describe makes a lot of sense, if you have independent priors (and likelihoods - you also get problems if observations in the two groups influence each other or are somehow correlated). A simple test for whether you really think that is to consider whether what you think about each group has nothing to do with what you see for the other group. E.g. if I show you one of the groups first, this will not change in the slightest what you expect for the other group.

To take an example, let's assume our outcome in a randomized controlled trial is a count outcome, we assume it's a Poisson distribution and we followed 100 patients on placebo for 1 year and we saw 4000 events. A-priori, we were pretty unsure about the placebo event rate and had a marginal Gamma(0.1,0.1) prior for both treatment groups. A-posteriori we now have a Gamma(4000.1, 100.1) prior for the annualized event rate. Does this change what you expect for the active treatment group? If you have independent priors for the two treatment groups, it does not.

This approach has problems, if you are e.g. somewhat unsure about where each treatment group ends up (you could still have marginal Gamma(0.1,0.1) priors for both treatment groups), but you assume that relative rate reductions (or increases) compared with placebo beyond 90% are pretty unlikely. In that case it might be more plausible to assume a prior such as, say, a N(0, 1) on the log-rate-ratio for treatment vs. placebo (instead of assuming a prior on the treatment group rate, although this of course implies one). In that case, your belief about what you expect to see in the treatment group changes after seeing the placebo group data (it's now the posterior for the placebo log-rate with the added noise of the N(0,1) prior for the treatment effect).

If that latter option is what you believe a-priori, then the approach you describe is not so suitable. I would guess that this is how many people in the clinical trials space think (i.e. they'd prefer to assign a prior on the expected placebo outcome and a prior on the treatment effect relative to that).

• That makes sense. I wasn’t even considering this possibility of putting a prior on a combination of the parameters, e.g., on their log ratio. In this setting, it’s clear that one would not be able to follow the steps I outlined (unless one marginalizes out the other dimension, thereby going back to the independent-priors scenario). What throws me off balance a bit is “if I show you one of the groups first, this will not change in the slightest what you expect for the other.” It feels like seeing results for one group always informs you about the other, because it is the same population.
– Ivan
Aug 6 at 7:37
• If you feel that one group tells you about the other, then independent priors are not right. Personally, I feel the same and would typically put priors on one group (e.g. the control group for which there's usually a lot of historical data) and then a prior on the treatment effect, as described. Aug 7 at 8:43

I don't have the reputation points to respond as a comment to another answer, but intuitively, what you are doing is fine if you assume there is no relationship between group A and group B.

How could such an assumption show up in your model? It would almost certainly be in the prior distribution. For a proportion, let's say you have 500 successes in 1000 trials in Group A, and 100 successes in 1000 trials in Group B.

If you assume independent Beta(1,1) priors (that's what the other posters means when they say "factorizing"), then the posteriors are Beta(501, 1001) for Group A and Beta(101, 10001) for Group B.

But you could also imagine a scenario in which you have little prior information, expect little data, and expect your two groups to be somewhat similar. Maybe it would make sense to utilize the information in the other group to inform the posterior. For instance, since we observed 50% success in Group A, and 10% success in Group B, we could say "I expected Group A to be around 40% and Group B to be around 20%", where we move our estimates towards the average of the two groups, 600 successes in 2000 trials = 30%. If your prior distributions are like this, then your estimates for theta_A and theta_B will correlated, and you can't independently sample like you want, because you won't be properly accounting for that correlation.

I assume you didn't do this for your priors, so I don't think you will have anything to worry about.

• Great point! Shrinkage via a multilevel setup is another prominent use case for having a prior that would not factorize. Indeed, if data is scarce, one would very much like to benefit from all data points and to see estimates pulled toward regions where there is more evidence
– Ivan
Aug 7 at 5:32

Let $$Y_A$$ and $$Y_B$$ denote the datasets for groups $$A$$ and $$B$$. Similarly let $$\theta_A$$ and $$\theta_B$$ denote the corresponding parameters.

If the joint distribution factors, $$$$p(Y_A,Y_B,\theta_A,\theta_B) = p(Y_A,\theta_A)\,p(Y_B,\theta_B) ,$$$$ then the joint posterior factors, $$$$p(\theta_A,\theta_B|Y_A,Y_B) = p(\theta_A|Y_A)\,p(\theta_B|Y_B) ,$$$$ where $$$$p(\theta_i|Y_i) = \frac{p(Y_i|\theta_i)\,p(\theta_i)}{p(Y_i)}$$$$ for $$i \in \{A,B\}$$.

The joint distribution will factor if the likelihood factors, $$$$p(Y_A,Y_B|\theta_A,\theta_B) = p(Y_A|\theta_A)\,p(Y_B|\theta_B) ,$$$$ and the prior factors, $$$$p(\theta_A,\theta_B) = p(\theta_A)\,p(\theta_B) .$$$$

If these assumptions are reasonable, then the proposed procedure is fine.

• Thank you! I suppose, if the randomization was done right, these are fair assumptions to make. Would you agree?
– Ivan
Jul 29 at 6:09
• @Ivan I'm not sure exactly what "the randomization" means here, but I suspect that's right with respect to the likelihood. But for the prior it depends on whether knowing one of the parameters tells you anything about the other one before you see the data. It's often assumed that it doesn't.
– mef
Jul 29 at 18:02
• I am referring to a randomized controlled trial (RCT) whose execution is sound, including random treatment assignment. I miss this connection in the answer. I would appreciate if you could add a sentence or two explaining your thoughts on the factorization of the prior of the parameters in a RCT.
– Ivan
Jul 30 at 5:42