# Comparing differences in frequencies

Let's say that we have four groups of individuals, namely groups A, B, C and D, randomly sampled from a greater population (in the underlying population, each individual also belongs to one of those four groups, necessarily).

Those four samples respectively have known sample sizes $$n_A, \ldots, n_D$$. In each sample, the relative frequency of a given trait has been measured: a proportion $$p_A$$ of individuals from sample A have this trait, ..., and a proportion $$p_D$$ in sample D.

Among other research questions, we would like to say whether the difference between $$p_A$$ and $$p_B$$ is significantly greater than the difference between $$p_C$$ and $$p_D$$. (I.e., we would like to know whether it is reasonable to accept the hypothesis that we have $$p_A - p_B > p_C - p_D$$ in the underlying population.)

What would be the appropriate statistical approach?

Thanks!

Given that the samples are independent, I would

1. Infer the posterior of the binomial rate in each sample, e.g., $$p_A n_A \sim Binomial(n_A, p_A)$$.
2. Compute the posterior distribution for ($$\delta_{AB} = p_A - p_B$$ and $$\delta_{CD} = p_C - p_D$$).
3. Compute the posterior distribution for $$\Delta = \delta_{AB} - \delta_{CD}$$.
4. The evidence that $$\delta_{AB} > \delta_{CD}$$ is now the area of the posterior where $$\Delta > 0$$.

Because I am lazy, I would use MCMC sampling for each of the binomial rates and subtract pairwise iterations for each of the parameters. The evidence is then the proportion of samples where $$\Delta > 0$$.

If you don't find anything better and your samples are not too small, you can use normal approximations. $$\hat p_A$$ will be approximately normal with mean $$p_A$$ and variance $$\frac{p_A(1-p_A)}{n}$$, analogously for the others. All of these assumed independent (also group members need to be assumed independent or each other), $$\hat p_A-\hat p_B$$ will have mean $$p_A-p_B$$ and variance $$\frac{p_A(1-p_A)}{n_A}+\frac{p_B(1-p_B)}{n_B}$$, analogously for $$p_C-p_D$$. With this you have a testing problem of whether two normal means are the same, where the variances are different.

$$(\hat p_A-\hat p_B)-(\hat p_C-\hat p_D)$$ is still approximately normal (all $$n_A$$ to $$n_D$$ not too small assumed) and has variance $$V=\frac{p_A(1-p_A)}{n_A}+\frac{p_B(1-p_B)}{n_B}+\frac{p_C(1-p_C)}{n_C}+\frac{p_D(1-p_D)}{n_D}$$. So under null hypothesis $$\frac{(\hat p_A-\hat p_B)-(\hat p_C-\hat p_D)}{\sqrt{\hat V}}$$, where $$\hat V$$ is the variance above with $$p_A$$ to $$p_D$$ replaced by the estimators, should be approximately $${\cal N}(0,1)$$-distributed, from which you can construct approximate one- or two-sided tests.

PS: In my notation the hat denotes estimators computed from the data, whereas without hat it's the theoretical quantities you're estimating. This is seemingly not totally in line with your notation, but more standard, and it's generally useful to distinguish estimators computed on data from what is estimated.