How to hypothesis test week-over-week metric for "significance", contrasting between test and control groups?

Question

I have conducted an experiment to test a hypothesis that changing the layout of my website would increase overall user engagement with the website. I therefore assigned 50% of the website visitors to see the "regular" layout (i.e., Control group) whereas the other 50% would see the new layout (i.e., Experiment group).

To measure the user engagement in each group, I chose the week-over-week metric: how many – out of those visiting the website in week i – have re-visited in week i+1.

Therefore, to conclude whether my layout change increases user engagement, my independent variable is Group (Experiment/Control) and dependent variable is $$ \frac{visitor-count-of-those-who-visited-in-both-week-i-AND-week-(i+1)}{visitor-count-of-those-who-visited-in-week-(i)} $$

I let the experiment run for 5 weeks. This means that I have 4 pairs of weeks as my data:

• week 0 -> week 1
• week 1 -> week 2
• week 2 -> week 3
• week 3 -> week 4

My question is – how should I test the difference between Control vs. Experiment groups?

---

Reproducible Example

Let's consider the following data as the results of my experiment:

pair_of_weeks	group	user_count_week_i	user_count_both_week_i_and_week i+1
0->1	control	3774	3169
0->1	experiment	3580	3031
1->2	control	4722	3661
1->2	experiment	4526	3609
2->3	control	5099	3790
2->3	experiment	4968	3746
3->4	control	5130	3810
3->4	experiment	4985	3792

Calculating the Week-Over-Week per each pair_of_weeks using R:

library(tibble)
library(dplyr)
library(ggplot2)
library(scales)

df <-
  tibble::tribble(
    ~pair_of_weeks,       ~group, ~user_count_week_i, ~user_count_both_week_i_and_week_i_plus_one,
    "0->1",    "control",              3774L,                                  3169L,
    "0->1", "experiment",              3580L,                                  3031L,
    "1->2",    "control",              4722L,                                  3661L,
    "1->2", "experiment",              4526L,                                  3609L,
    "2->3",    "control",              5099L,                                  3790L,
    "2->3", "experiment",              4968L,                                  3746L,
    "3->4",    "control",              5130L,                                  3810L,
    "3->4", "experiment",              4985L,                                  3792L
  )

df |> 
  mutate(week_over_week_prop = user_count_both_week_i_and_week_i_plus_one / user_count_week_i) |> 
  mutate(labeling = paste0(percent(week_over_week_prop, 0.1), 
                           "\n",
                           "(", 
                           comma(user_count_week_i),
                           "->", 
                           comma(user_count_both_week_i_and_week_i_plus_one), 
                           ")"
  )
  ) |> 
  ggplot(aes(x = pair_of_weeks, y = week_over_week_prop, fill = group)) +
  geom_col(position = position_dodge(width = 0.8))  +
  geom_text(aes(label = labeling), position = position_dodge(width = 0.8), vjust = -0.1, size = rel(3)) +
  expand_limits(y = c(0, 1) )

---

Now, what if I want to conclude whether the difference between Control vs Experiment groups is "significant"? One way is to examine each pair of weeks:

library(broom)

week_0_1 <- prop.test(x = c(3169, 3031), n = c(3774, 3580)) |> broom::tidy()
week_1_2 <- prop.test(x = c(3661, 3609), n = c(4722, 4526)) |> broom::tidy()
week_2_3 <- prop.test(x = c(3790, 3746), n = c(5099, 4968)) |> broom::tidy()
week_3_4 <- prop.test(x = c(3810, 3792), n = c(5130, 4985)) |> broom::tidy()

bind_rows(week_0_1,
          week_1_2,
          week_2_3,
          week_3_4) |> 
  tibble::add_column(pair_of_weeks = c("0-1", "1-2", "2-3", "3-4"), .before = 0)
#> # A tibble: 4 × 10
#>   pair_of_weeks estimate1 estimate2 statistic p.value parameter conf.low
#>   <chr>             <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>
#> 1 0-1               0.840     0.847     0.620  0.431          1  -0.0239
#> 2 1-2               0.775     0.797     6.57   0.0104         1  -0.0390
#> 3 2-3               0.743     0.754     1.49   0.223          1  -0.0279
#> 4 3-4               0.743     0.761     4.29   0.0384         1  -0.0350
#> # ℹ 3 more variables: conf.high <dbl>, method <chr>, alternative <chr>

If we use the pval of 0.05 as the threshold for "significance", we can see that only pairs of weeks "1-2" and "3-4" are below pval of 0.05.

Therefore I have no decisive conclusion. Is there another way for me to test the results, overall, for the entire span of the experiment (as opposed to breaking down to pairs of weeks)?

Also important to mention that there is a heavy, natural, overlap in the identity of the visitors/users of the webpage, both within and between pairs of weeks.

Answer 1

A few comments and a few suggestions; hopefully this helps you find the proper answer.

1. The simplest, most direct answer to your question ("Is there another way for me to test the results, overall, for the entire span of the experiment") is to aggregate all your results: the numerator will be the sum of all your numerators, over the 4 weekly comparisons, and the denominator will equally be the sum of all 4 denominators, over the 4 weekly comparisons. One comparison, of 2 proportions between Control and Experimantal groups, which is either significant or not. Easy. But this would be the wrong answer, as I will try to explain below.
2. There seems to be no clear reason why you are looking "week-to-week". This 1 week lag seems quite arbitrary; what if a visitor comes back the next day, or even the same day? Is that not "engagement". What if a visitor comes back 2 weeks later; is that not also "engagement"? And what if a visitor comes back 6 times over the next 5 weeks? Is that not more "engagement" than the visitor who came back only once over the 5 weeks? It also not as if you tried a new web design each week; each week is not a new experiement. You. have a single experiment which lasted 5 weeks.
3. Based on this, your numerator should be the count of repeat visits over the full 4 weeks. If visitor A visited 8 times over the 5 weeks, this should add 7 to your numerator (which is the count of repeat visits over the 5 weeks; a visitor with only a single visit would add 0 - clearly he/she was not "hooked", while a visitor who came 8 times adds 7 -clearly liked what he/she saw).
4. It is not clear if the visitors in your denominator are really unique visitors, or total visits. That is, if visitor $B$ came 4 times in week $i$, does he/she add 4 to the denominator or only 1. It should be 1. The denominator should be the count of unique visitors over the 5 weeks, because your experimental unit is a visitor (not a visit).
5. So your metric becomes $\frac {count-of-repeat-visits-over-5-weeks} {count-of-unique-visitors-over-the-5-weeks}$. You can run the same test, and will get a single, aggregate answer for the whole 5 weeks experiment.
6. But unfortunately, while this should be a better approach, it is again too simple. It is very likely (as you yourself write: "Also important to mention that there is a heavy, natural, overlap in the identity of the visitors/users of the webpage, both within and between pairs of weeks."), that most visitors in your experiment are already regular visitors, who previously visited; so in a way, they are already "engaged", are coming for the content, and are unlikely to be swayed one way or the other by the design. One could even argue that "regular" visitors, who got used to the "old" design (no matter how poor), would dislike the new design, because now they have to figure it out (they need to re-learn to navigate it). They are not the right test subjects for your experiment. You should use "virgin" visitors, or as "virgin as possible" visitors (e.g. visitors who had not visited for the past $m$ months, or $w$ weeks, etc.), to have a fair comparison of the design.
   Now, you may argue that in fact you want to see if "regular" visitors become even more engaged; then you should make sure to include only "regulars" (e.g. who visited at least $x$ times over the previous $y$ weeks before the start of the experiment). The point being that virgin visitors, and regular visitors are 2 very distinct populations, and you need to be very clear which you want to test.
   You may even want to test both; i.e. compute the metric as described above for "virgins", and "regulars". You will probably find out that the results are quite different...