A common problem in tech-startups is cohort analysis when A/B testing new app features: within a certain time period between two different dates, two different sets of users joining between those dates experience two different versions of the app (A and B). In order to decide which version is better, the statistic we collect is the number of users $N_{t_0,\Delta t}$ which joined on day $t_0$ and used the app $\Delta t$ days later. The data typically looks likes this

Group A

Cohort  Day0    Day1    Day2    Day3
25 July 2614    351     140     152
26 July 2819    571     210
27 July 2261    415

Group B

Cohort  Day0    Day1    Day2    Day3
25 July 2608    411     151     148
26 July 2822    592     264
27 July 2301    444

Now we would like to run a significance test to find out whether one feature is better than the other or not. The way I did this was to run Fisher's exact test for each $(t_0, \Delta t)$ pair by constructing the corresponding contingency table and obtain a p-value, and then combine all of the p-values using Fisher's method.

The result looks like this:

Testing Group A BETTER AT TIMES than Group B:

Cohort 1, day 1 test p-value: 0.992
Cohort 1, day 2 test p-value: 0.772
Cohort 1, day 3 test p-value: 0.437
Cohort 2, day 1 test p-value: 0.759
Cohort 2, day 2 test p-value: 0.996
Cohort 3, day 1 test p-value: 0.803

Combined p-value: 0.994

Testing if Group A WORSE AT TIMES than Group B:

Cohort 1, day 1 test p-value: 0.009
Cohort 1, day 2 test p-value: 0.267
Cohort 1, day 3 test p-value: 0.609
Cohort 2, day 1 test p-value: 0.262
Cohort 2, day 2 test p-value: 0.006
Cohort 3, day 1 test p-value: 0.219

Combined p-value: 0.004

Does this seem like a reasonable way to do this analysis and decide significance? We want to write an article and share the code, and would like to make sure I'm not doing anything wrong. (One thing I'd like to note here is that it's not quite survival analysis, as the number of users in a cohort can actually increase from one day to another, which I have observed in real data)


1 Answer 1


You have approached the problem by performing two one-sided tests of proportions; A > B and B > A. This is clearly problematic. You should ideally posit one ahead of time before looking at the data then test that if you choose to use a one-sided tests. Imagine if ahead of time, you had posited A > B, then nothing would be statistically significant. So most investigators use a single two-sided of any difference between A and B. It is possible to perform a two-sided test using Fisher's exact test.

During testing, you have performed comparisons within cohorts. Is this really necessary? Your main interest is which is better: A or B? Unless you have reason to believe each cohorts represents unique information, it is perhaps more reasonable to perform the analysis across cohorts, but possibly accounting for them in some way. If you specifically want to test within cohorts, then sure you can. But if the insights are not so different, then you will be creating more results than you have to, and repeating the story over and over.

Finally, over the use of Fisher's exact test. At a large sample size such as yours, there is no need to use Fisher's exact test. Large sample methods such as logistic regression fitted using maximum likelihood should be just fine. More importantly, an approach that is readily extensible is preferable. So instead of doing several two-way comparisons using specialized methods like Fisher's exact test, you can run a single regression model where everything that is relevant is accounted for in one model.

With that said, here's how I might analyze the data. First, I reorganized it to have a column for group, cohort, number of persons on day 0, day (1-3), and the number of people who used the site.

So the data now look like:

   grp cohort total   day people
1    A     C1  2614 day_1    351
2    A     C2  2819 day_1    571
3    A     C3  2261 day_1    415
4    B     C1  2608 day_1    411
5    B     C2  2822 day_1    592
6    B     C3  2301 day_1    444
7    A     C1  2614 day_2    140
8    A     C2  2819 day_2    210
10   B     C1  2608 day_2    151
11   B     C2  2822 day_2    264
13   A     C1  2614 day_3    152
16   B     C1  2608 day_3    148

My approach is to think of each row as n Bernoulli attempts to visit your website and where n is the value of total, and people is the number of actually show up (successes). So total - people would be the number of failures.

Phrased this way, I can model the problem using logistic regression. As predictors, I can use the different days as I see that the probability of success drops drastically from day 1 to day 2 and then appears to be steady. Additionally, I am interested in how this probability is different depending on group membership, A or B. I can run this analysis cohort by cohort:

glm( # Cohort 1
  cbind(people, total - people) ~ 0 + day + day:grp, binomial, dat,
  subset = cohort == "C1")
glm( # Cohort 2
  cbind(people, total - people) ~ 0 + day + day:grp, binomial, dat,
  subset = cohort == "C2")
glm( # Cohort 3, only one day so a simple group difference
  cbind(people, total - people) ~ grp, binomial, dat,
  subset = cohort == "C3")

Alternatively, I can run this analysis assuming the group differences are the same across cohorts In R, the syntax given the dataset would then be:

mod.c <- glm(
  cbind(people, total - people) ~ 0 + day + cohort + day:grp,
  binomial, dat)

I passed a two-column matrix with the first column representing the number of successes and the second, the number of failures. I used 0 + to suppress the intercept. This way, I get a coefficient for each day. I also added the interaction between day and group. By alphabetical order, group A becomes my reference group and the interaction represents the deviation of group B from group A on each day.

Let's see the results:

round(coef(summary(mod.c)), 5)
              Estimate Std. Error   z value Pr(>|z|)
dayday_1      -1.81691    0.04161 -43.66453  0.00000
dayday_2      -2.91613    0.06114 -47.69310  0.00000
dayday_3      -2.78485    0.08358 -33.32074  0.00000
cohortC2       0.42569    0.04264   9.98296  0.00000
cohortC3       0.30949    0.05194   5.95825  0.00000
dayday_1:grpB  0.09051    0.04200   2.15514  0.03115
dayday_2:grpB  0.18399    0.07536   2.44146  0.01463
dayday_3:grpB -0.02586    0.11895  -0.21737  0.82792

First, as to statistical significance. The difference between both groups was statistically significant on days 1 and 2 with group B being higher. Group B was lower on day 3, but this difference was not statistically significant.

We can conduct a likelihood ratio test of all three group B deviations to see whether these deviations are worth considering in the model:

library(lmtest) # The lmtest package is handy here
lrtest(mod.c, "day:grp") # We test the interaction effect
Likelihood ratio test

Model 1: cbind(people, total - people) ~ 0 + day + cohort + day:grp
Model 2: cbind(people, total - people) ~ day + cohort - 1
  #Df  LogLik Df  Chisq Pr(>Chisq)  
1   8 -45.736                       
2   5 -51.074 -3 10.676    0.01362 *
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

The likelihood ratio test comparing the model with the interaction to one without suggests the group differences are worth considering in trying to understand these data, $\chi^2(3)=10.68, p = .014$. You can also do AIC and BIC comparisons.

The next step might be interpreting those coefficients. It is best to convert them to probabilities. First, I report the complete regression equation:

$y = -1.82 \times day1 - 2.92 \times day2 -2.78 \times day3 + 0.43 \times cohort2 + 0.31 \times cohort3 + 0.09 \times day1 \times groupB + 0.18 \times day2 \times groupB - 0.03 \times day3 \times groupB$, where the variables are indicators for the day and group.

So what is the $y$ for group A on day 1, cohort 1? We can use the regression equation. That would be -1.82. The same value for group B would be $-1.82 + 0.09=-1.73$. Next, we convert these values to probabilities using the formula, $(1+e^{-y})^{-1}$. We arrive at predicted probabilities of about 14% and 15% respectively.

Again, what is the $y$ for group A on day 2, cohort 3? We can use the regression equation. That would be $-2.92+0.31=-2.61$. The same value for group B would be $-2.92+0.31+0.18=-2.43$. Next, we convert these values to probabilities using the formula, $(1+e^{-y})^{-1}$. We arrive at predicted probabilities of about 6.8% and 8.1% respectively.

We can take the differences to arrive at the difference in probabilities between both groups on each of these days within cohorts. The statistical tests from earlier already tell us which days the differences in group probability of success are statistically different. We also know that overall, these group differences are worth considering.

When making predictions, you should probably not round your coefficients to two decimals places first, as you introduce error due to rounding. Use the full 8/9 digit coefficient reported by software to predict y, then transform to a probability, which you can then report to two decimal places if you want to.

  • 1
    $\begingroup$ Thanks for contributing. I am aware that there are multiple ways to do it, but my question is about whether the way I presented is incorrect in any way $\endgroup$
    – Gabi
    Jul 30, 2018 at 12:12
  • $\begingroup$ If another method is clearly better, that would be a great find, but we'd need to understand why a different proposed method is strictly better than the one I presented. $\endgroup$
    – Gabi
    Jul 30, 2018 at 12:18
  • 1
    $\begingroup$ @Gabi I have modified my response. I now hope it addresses your question. $\endgroup$ Jul 30, 2018 at 13:28
  • $\begingroup$ Thanks a lot! I was going to respond in detail, but then realized stack exchange doesn't let me (I'm new here). Please let me know if there's any way I could do that if you'd like me to :) $\endgroup$
    – Gabi
    Aug 2, 2018 at 9:28
  • $\begingroup$ @Gabi I think we can move comments to chat. $\endgroup$ Aug 5, 2018 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.