0
$\begingroup$

I'm trying to evaluate whether the difference in uplift seen in below table between the test groups and the control group is statistically significant. I'm unsure about the appropriate statistical test.

The test and control groups are all of different sizes already before the test, which is why I have included the relative numbers. I first thought about the chi-square test, but that don't think it captures the pre- & post-treatment aspect correctly.

For context: Four different geographic regions were selected, one of them as control. Each test region received a different mix of marketing measures with the goal to raise awareness. I am now trying to evaluate whether the uplift in the test regions is statistically different from the control region.

Control Group 1 Group 2 Group 3
Number of visitors (pretest) 59800 9993 19284 17876
Number of visitors (posttest) 65993 11781 23373 20883
Relative Change +10.36% +17.89% +21.20% +16.82%

Which statistical test would be needed to find an answer to my question?

Thank you very much.

Edit:

The below table shows the weekly visitors by test group. Week 23 to 28 are pre-treatment, week 29 to 34 are post-treatment.

Control Group 1 Group 2 Group 3
Week 23 8590 1492 2929 2837
Week 24 9217 1588 3138 2846
Week 25 9534 1599 2992 2812
Week 26 10213 1714 3440 3005
Week 27 10435 1704 3187 2987
Week 28 10932 1817 3180 3234
Week 29 11489 1948 3566 3159
Week 30 10936 1974 3707 3273
Week 31 11856 2061 3885 3609
Week 32 10621 1851 3926 3586
Week 33 9905 1852 4051 3372
Week 34 9812 1850 3621 3203
$\endgroup$
6
  • $\begingroup$ Can you first explain some more about your experiment? What was your goal, how was group membership determined, etc $\endgroup$ Aug 31, 2023 at 12:09
  • $\begingroup$ Do you have e.g. daily, weekly or monthly visitor counts for some time period pre- and posttest, or only these overall counts? $\endgroup$
    – Sointu
    Aug 31, 2023 at 12:14
  • $\begingroup$ @DemetriPananos Our goal was to increase brand awareness. The three test groups received different amounts of budget and different awareness measures were used per group. The groups just relate to different federal states, data is available only on group level. The variable "visitors" is a proxy measure to determine if the increased awareness spendings and particular mix caused an uplift in website visitors. $\endgroup$
    – Liam K
    Aug 31, 2023 at 12:28
  • $\begingroup$ @Sointu Yes, I can access data on any level, i.e. daily or weekly. $\endgroup$
    – Liam K
    Aug 31, 2023 at 12:30
  • $\begingroup$ Can we get the data on a weekly level? $\endgroup$ Aug 31, 2023 at 12:51

1 Answer 1

0
$\begingroup$

This probably isn't the best approach, but I anticipate you don't need best, you just need "good enough".

Given your control isn't a typical "control" in the sense of a randomized experiment, you should use something called a "Difference in Difference" approach.

Basically, fit the following model

$$ y_i = \beta_0 + \beta_1 t + \beta_2 x + \beta_3 t \cdot x + \epsilon_i $$

  • $\beta_0$ is the average number of visitors in the control group in the pre-period
  • $\beta_1$ is difference in average visitors in the control between pre and post periods
  • $\beta_2$ is the difference in average visitors between control and group $i$ in the pre period, and
  • $\beta_3$ is the difference in group $i$ between how they actually evolved in time and had they evolved like control. This relies on the parallel trends assumption.

I'm not going to give an entire lesson on difference in difference, but it would be a good idea for you to review the method, interpretation, and assumptions made therein.

Assuming this is a valid approach (I'll comment on why it might not be and why you might still use it anyway a little later), we can estimate the differences for all groups at the same time. Using R...

library(tidyverse)

data.frame(
  stringsAsFactors = FALSE,
                time = c("Week 23","Week 24","Week 25",
                       "Week 26","Week 27","Week 28","Week 29","Week 30",
                       "Week 31","Week 32","Week 33","Week 34"),
                control = c(8590L,9217L,9534L,10213L,
                       10435L,10932L,11489L,10936L,11856L,10621L,9905L,
                       9812L),
                G1 = c(1492L,1588L,1599L,1714L,
                       1704L,1817L,1948L,1974L,2061L,1851L,1852L,1850L),
                G2 = c(2929L,3138L,2992L,3440L,
                       3187L,3180L,3566L,3707L,3885L,3926L,4051L,3621L),
                G3 = c(2837L,2846L,2812L,3005L,
                       2987L,3234L,3159L,3273L,3609L,3586L,3372L,3203L)
) -> d


md <- d %>% 
  mutate(t = rep(0:1, each=6)) %>% 
  pivot_longer(control:G3, names_to = 'group', values_to = 'y')

fit <- lm(y ~ t*group, data=md)
summary(fit)

Call:
lm(formula = y ~ t * group, data = md)

Residuals:
     Min       1Q   Median       3Q      Max 
-1230.17  -149.71    -0.67   144.92  1111.83 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   9820.2      181.4  54.133  < 2e-16 ***
t              949.7      256.6   3.702 0.000645 ***
groupG1      -8167.8      256.6 -31.837  < 2e-16 ***
groupG2      -6675.8      256.6 -26.021  < 2e-16 ***
groupG3      -6866.7      256.6 -26.765  < 2e-16 ***
t:groupG1     -679.3      362.8  -1.872 0.068477 .  
t:groupG2     -301.3      362.8  -0.831 0.411166    
t:groupG3     -536.2      362.8  -1.478 0.147297    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 444.4 on 40 degrees of freedom
Multiple R-squared:  0.9853,    Adjusted R-squared:  0.9827 
F-statistic: 382.6 on 7 and 40 DF,  p-value: < 2.2e-16

What you care about are the coefficients that start with t:group. Those are the $\beta_3$ type coefficients for each group. Associated estimates and p values are shown there as well.

There is probably a better way to do this using fixed effect estimation, but I don't think you need that and this might be good enough for your purposes.

$\endgroup$
3
  • $\begingroup$ Thank you very much. That helps a lot. Although not the result I was hoping for, but that's just hypothesis testing :) Also, thank you for introducing me to the concept of "differences in differences approach", quite interesting. $\endgroup$
    – Liam K
    Sep 1, 2023 at 12:20
  • $\begingroup$ One more question, what does the t-value express in the coefficients table? $\endgroup$
    – Liam K
    Sep 1, 2023 at 12:24
  • $\begingroup$ @LiamK The t value are the test statistics for the coefficients. They are used to calculate the associated p value. For more, I would learn about hypothesis testing via linear regression. $\endgroup$ Sep 1, 2023 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.