# Compare daily Control-to-Test uplifts of two Groups

I need to analyze app-campaigns and compare the proportion of daily active users to the group size , in a control group (CG) and a Test group (TG). I always create a control group of users who were targeted according to the same criteria, but never get the in-app campaign.

Let

• $X_{CG}$ be the daily number of active users in the CG
• $X_{TG}$ be the daily number of active users in the TG.

Also, let

• $N_{CG}$ be the group size of the Control group
• $N_{TG}$ be the group size of the Test Group.

Let $\Delta=\frac{X_{TG}}{N_{TG}} - \frac{X_{CG}}{N_{CG}}$ be the difference of proportions between the TG and the CG. I use a one sided test on proportions to check either the proportion of daily active users in the TG becomes higher than in the CG (due to the campaign), that is: $$H_0: \Delta=0 \\ H_1: \Delta>0$$ Then, using the normal approximation of the binomial distribution, it can be written that, under the null hypothesis that both proportions are equal, and with a hypothesis of homo-schedasticity:

$$\Delta \sim_{H_0} N(0,p \times q \times(\frac{1}{N_{TG}}+\frac{1}{N_{CG}}))$$

where p is the pooled probability of outcome defined as $\frac{X_{TG}+X_{CG}}{N_{TG}+N_{CG}}$ and $q=1-p$.

The above test on proportion is classical. Now comes the core of my question. Suppose that now, I have two sets of control group and test group, let's say $(CG_1,TG_1)$ and $(CG_2,TG_2)$, with

• $\Delta_1=\frac{X^1_{TG}}{N^1_{TG}} - \frac{X^1_{CG}}{N^1_{CG}}$
• $\Delta_2=\frac{X^2_{TG}}{N^2_{TG}} - \frac{X^2_{CG}}{N^2_{CG}}$

I would like to statistically compare $\Delta_1$ and $\Delta_2$ of the two sets, by checking the statistical significance of the difference $\Delta_1-\Delta_2$. Specifically, I want to carry-out the following two sided-test:

• $H0: \Delta_1=\Delta_2$
• $H1: \Delta_1≠\Delta_2$

Under the normal approximation on proportions, what could be the used statistic for such a test?

I thought about the following distribution. under the null hypothesis of equality of uplifts: $$\Delta_1 - \Delta_2 \sim N \bigg( 0,p_1 \times q_1 \times (\frac{1}{N^1_{CG}}+\frac{1}{N^1_{TG}})+p_2 \times q_2 \times (\frac{1}{N^2_{CG}}+\frac{1}{N^2_{TG}}) \bigg)$$

Where

• $N^1_{CG}$ is the group size of the control group of Set 1.
• $N^1_{TG}$ is the group size of the test group of Set 1.
• $N^2_{CG}$ is the group size of the control group of Set 2.
• $N^2_{TG}$ is the group size of the test group of Set 2.
• $p_1$ an $p_2$ the pooled probabilities of outcome and finally, $q_1=1-p_1$ and $q_2=1-p_2$.

The distribution you thought about is correct. Since the difference of two independent normal random variables is again normal with mean $\mu = \mu_1 - \mu_2$ and $\sigma^2=\sigma_1^2 + \sigma_2^2$. See this answer for reference.
However, this only holds if $\Delta_1$ and $\Delta_2$ are independent. This should be the case in practice if $CG_1, CG_2, TG_1, TG_2$ are mutually exclusive. As soon as you start having some overlaps you need to be careful.