Appropriate statistical test to determine if uplift between control group and multiple test groups is significant (pretest/posttest evaluation)

Question

I'm trying to evaluate whether the difference in uplift seen in below table between the test groups and the control group is statistically significant. I'm unsure about the appropriate statistical test.

The test and control groups are all of different sizes already before the test, which is why I have included the relative numbers. I first thought about the chi-square test, but that don't think it captures the pre- & post-treatment aspect correctly.

For context: Four different geographic regions were selected, one of them as control. Each test region received a different mix of marketing measures with the goal to raise awareness. I am now trying to evaluate whether the uplift in the test regions is statistically different from the control region.

	Control	Group 1	Group 2	Group 3
Number of visitors (pretest)	59800	9993	19284	17876
Number of visitors (posttest)	65993	11781	23373	20883
Relative Change	+10.36%	+17.89%	+21.20%	+16.82%

Which statistical test would be needed to find an answer to my question?

Thank you very much.

Edit:

The below table shows the weekly visitors by test group. Week 23 to 28 are pre-treatment, week 29 to 34 are post-treatment.

	Control	Group 1	Group 2	Group 3
Week 23	8590	1492	2929	2837
Week 24	9217	1588	3138	2846
Week 25	9534	1599	2992	2812
Week 26	10213	1714	3440	3005
Week 27	10435	1704	3187	2987
Week 28	10932	1817	3180	3234
Week 29	11489	1948	3566	3159
Week 30	10936	1974	3707	3273
Week 31	11856	2061	3885	3609
Week 32	10621	1851	3926	3586
Week 33	9905	1852	4051	3372
Week 34	9812	1850	3621	3203

Answer 1

This probably isn't the best approach, but I anticipate you don't need best, you just need "good enough".

Given your control isn't a typical "control" in the sense of a randomized experiment, you should use something called a "Difference in Difference" approach.

Basically, fit the following model

$$ y_i = \beta_0 + \beta_1 t + \beta_2 x + \beta_3 t \cdot x + \epsilon_i $$

• $\beta_0$ is the average number of visitors in the control group in the pre-period
• $\beta_1$ is difference in average visitors in the control between pre and post periods
• $\beta_2$ is the difference in average visitors between control and group $i$ in the pre period, and
• $\beta_3$ is the difference in group $i$ between how they actually evolved in time and had they evolved like control. This relies on the parallel trends assumption.

I'm not going to give an entire lesson on difference in difference, but it would be a good idea for you to review the method, interpretation, and assumptions made therein.

Assuming this is a valid approach (I'll comment on why it might not be and why you might still use it anyway a little later), we can estimate the differences for all groups at the same time. Using R...

library(tidyverse)

data.frame(
  stringsAsFactors = FALSE,
                time = c("Week 23","Week 24","Week 25",
                       "Week 26","Week 27","Week 28","Week 29","Week 30",
                       "Week 31","Week 32","Week 33","Week 34"),
                control = c(8590L,9217L,9534L,10213L,
                       10435L,10932L,11489L,10936L,11856L,10621L,9905L,
                       9812L),
                G1 = c(1492L,1588L,1599L,1714L,
                       1704L,1817L,1948L,1974L,2061L,1851L,1852L,1850L),
                G2 = c(2929L,3138L,2992L,3440L,
                       3187L,3180L,3566L,3707L,3885L,3926L,4051L,3621L),
                G3 = c(2837L,2846L,2812L,3005L,
                       2987L,3234L,3159L,3273L,3609L,3586L,3372L,3203L)
) -> d


md <- d %>% 
  mutate(t = rep(0:1, each=6)) %>% 
  pivot_longer(control:G3, names_to = 'group', values_to = 'y')

fit <- lm(y ~ t*group, data=md)
summary(fit)

Call:
lm(formula = y ~ t * group, data = md)

Residuals:
     Min       1Q   Median       3Q      Max 
-1230.17  -149.71    -0.67   144.92  1111.83 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   9820.2      181.4  54.133  < 2e-16 ***
t              949.7      256.6   3.702 0.000645 ***
groupG1      -8167.8      256.6 -31.837  < 2e-16 ***
groupG2      -6675.8      256.6 -26.021  < 2e-16 ***
groupG3      -6866.7      256.6 -26.765  < 2e-16 ***
t:groupG1     -679.3      362.8  -1.872 0.068477 .  
t:groupG2     -301.3      362.8  -0.831 0.411166    
t:groupG3     -536.2      362.8  -1.478 0.147297    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 444.4 on 40 degrees of freedom
Multiple R-squared:  0.9853,    Adjusted R-squared:  0.9827 
F-statistic: 382.6 on 7 and 40 DF,  p-value: < 2.2e-16

What you care about are the coefficients that start with t:group. Those are the $\beta_3$ type coefficients for each group. Associated estimates and p values are shown there as well.

There is probably a better way to do this using fixed effect estimation, but I don't think you need that and this might be good enough for your purposes.