1
$\begingroup$

I need to analyze app-campaigns and compare the proportion of daily active users to the group size , in a control group (CG) and a Test group (TG). I always create a control group of users who were targeted according to the same criteria, but never get the in-app campaign.

Let

  • $X_{CG}$ be the daily number of active users in the CG
  • $X_{TG}$ be the daily number of active users in the TG.

Also, let

  • $N_{CG}$ be the group size of the Control group
  • $N_{TG}$ be the group size of the Test Group.

Let $\Delta=\frac{X_{TG}}{N_{TG}} - \frac{X_{CG}}{N_{CG}}$ be the difference of proportions between the TG and the CG. I use a one sided test on proportions to check either the proportion of daily active users in the TG becomes higher than in the CG (due to the campaign), that is: $$H_0: \Delta=0 \\ H_1: \Delta>0$$ Then, using the normal approximation of the binomial distribution, it can be written that, under the null hypothesis that both proportions are equal, and with a hypothesis of homo-schedasticity:

$$\Delta \sim_{H_0} N(0,p \times q \times(\frac{1}{N_{TG}}+\frac{1}{N_{CG}}))$$

where p is the pooled probability of outcome defined as $\frac{X_{TG}+X_{CG}}{N_{TG}+N_{CG}}$ and $q=1-p$.

The above test on proportion is classical. Now comes the core of my question. Suppose that now, I have two sets of control group and test group, let's say $(CG_1,TG_1)$ and $(CG_2,TG_2) $, with

  • $\Delta_1=\frac{X^1_{TG}}{N^1_{TG}} - \frac{X^1_{CG}}{N^1_{CG}}$
  • $\Delta_2=\frac{X^2_{TG}}{N^2_{TG}} - \frac{X^2_{CG}}{N^2_{CG}}$

I would like to statistically compare $\Delta_1$ and $\Delta_2$ of the two sets, by checking the statistical significance of the difference $\Delta_1-\Delta_2$. Specifically, I want to carry-out the following two sided-test:

  • $H0: \Delta_1=\Delta_2$
  • $H1: \Delta_1≠\Delta_2$

Under the normal approximation on proportions, what could be the used statistic for such a test?

I thought about the following distribution. under the null hypothesis of equality of uplifts: $$\Delta_1 - \Delta_2 \sim N \bigg( 0,p_1 \times q_1 \times (\frac{1}{N^1_{CG}}+\frac{1}{N^1_{TG}})+p_2 \times q_2 \times (\frac{1}{N^2_{CG}}+\frac{1}{N^2_{TG}}) \bigg)$$

Where

  • $N^1_{CG}$ is the group size of the control group of Set 1.
  • $N^1_{TG}$ is the group size of the test group of Set 1.
  • $N^2_{CG}$ is the group size of the control group of Set 2.
  • $N^2_{TG}$ is the group size of the test group of Set 2.
  • $p_1$ an $p_2$ the pooled probabilities of outcome and finally, $q_1=1-p_1$ and $q_2=1-p_2$.

I'd be glad to get your suggestions on this. Thanks in advance!

$\endgroup$
0
$\begingroup$

The distribution you thought about is correct. Since the difference of two independent normal random variables is again normal with mean $\mu = \mu_1 - \mu_2$ and $\sigma^2=\sigma_1^2 + \sigma_2^2$. See this answer for reference.

However, this only holds if $\Delta_1$ and $\Delta_2$ are independent. This should be the case in practice if $CG_1, CG_2, TG_1, TG_2$ are mutually exclusive. As soon as you start having some overlaps you need to be careful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.