You have approached the problem by performing two one-sided tests of proportions; A > B and B > A. This is clearly problematic. You should ideally posit one ahead of time before looking at the data then test that if you choose to use a one-sided tests. Imagine if ahead of time, you had posited A > B, then nothing would be statistically significant. So most investigators use a single two-sided of any difference between A and B. It is possible to perform a two-sided test using Fisher's exact test.
During testing, you have performed comparisons within cohorts. Is this really necessary? Your main interest is which is better: A or B? Unless you have reason to believe each cohorts represents unique information, it is perhaps more reasonable to perform the analysis across cohorts, but possibly accounting for them in some way. If you specifically want to test within cohorts, then sure you can. But if the insights are not so different, then you will be creating more results than you have to, and repeating the story over and over.
Finally, over the use of Fisher's exact test. At a large sample size such as yours, there is no need to use Fisher's exact test. Large sample methods such as logistic regression fitted using maximum likelihood should be just fine. More importantly, an approach that is readily extensible is preferable. So instead of doing several two-way comparisons using specialized methods like Fisher's exact test, you can run a single regression model where everything that is relevant is accounted for in one model.
With that said, here's how I might analyze the data. First, I reorganized it to have a column for group, cohort, number of persons on day 0, day (1-3), and the number of people who used the site.
So the data now look like:
grp cohort total day people
1 A C1 2614 day_1 351
2 A C2 2819 day_1 571
3 A C3 2261 day_1 415
4 B C1 2608 day_1 411
5 B C2 2822 day_1 592
6 B C3 2301 day_1 444
7 A C1 2614 day_2 140
8 A C2 2819 day_2 210
10 B C1 2608 day_2 151
11 B C2 2822 day_2 264
13 A C1 2614 day_3 152
16 B C1 2608 day_3 148
My approach is to think of each row as n Bernoulli attempts to visit your website and where n is the value of total, and people is the number of actually show up (successes). So total - people would be the number of failures.
Phrased this way, I can model the problem using logistic regression. As predictors, I can use the different days as I see that the probability of success drops drastically from day 1 to day 2 and then appears to be steady. Additionally, I am interested in how this probability is different depending on group membership, A or B. I can run this analysis cohort by cohort:
glm( # Cohort 1
cbind(people, total - people) ~ 0 + day + day:grp, binomial, dat,
subset = cohort == "C1")
glm( # Cohort 2
cbind(people, total - people) ~ 0 + day + day:grp, binomial, dat,
subset = cohort == "C2")
glm( # Cohort 3, only one day so a simple group difference
cbind(people, total - people) ~ grp, binomial, dat,
subset = cohort == "C3")
Alternatively, I can run this analysis assuming the group differences are the same across cohorts In R, the syntax given the dataset would then be:
mod.c <- glm(
cbind(people, total - people) ~ 0 + day + cohort + day:grp,
binomial, dat)
I passed a two-column matrix with the first column representing the number of successes and the second, the number of failures. I used 0 + to suppress the intercept. This way, I get a coefficient for each day. I also added the interaction between day and group. By alphabetical order, group A becomes my reference group and the interaction represents the deviation of group B from group A on each day.
Let's see the results:
round(coef(summary(mod.c)), 5)
Estimate Std. Error z value Pr(>|z|)
dayday_1 -1.81691 0.04161 -43.66453 0.00000
dayday_2 -2.91613 0.06114 -47.69310 0.00000
dayday_3 -2.78485 0.08358 -33.32074 0.00000
cohortC2 0.42569 0.04264 9.98296 0.00000
cohortC3 0.30949 0.05194 5.95825 0.00000
dayday_1:grpB 0.09051 0.04200 2.15514 0.03115
dayday_2:grpB 0.18399 0.07536 2.44146 0.01463
dayday_3:grpB -0.02586 0.11895 -0.21737 0.82792
First, as to statistical significance. The difference between both groups was statistically significant on days 1 and 2 with group B being higher. Group B was lower on day 3, but this difference was not statistically significant.
We can conduct a likelihood ratio test of all three group B deviations to see whether these deviations are worth considering in the model:
library(lmtest) # The lmtest package is handy here
lrtest(mod.c, "day:grp") # We test the interaction effect
Likelihood ratio test
Model 1: cbind(people, total - people) ~ 0 + day + cohort + day:grp
Model 2: cbind(people, total - people) ~ day + cohort - 1
#Df LogLik Df Chisq Pr(>Chisq)
1 8 -45.736
2 5 -51.074 -3 10.676 0.01362 *
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
The likelihood ratio test comparing the model with the interaction to one without suggests the group differences are worth considering in trying to understand these data, $\chi^2(3)=10.68, p = .014$. You can also do AIC and BIC comparisons.
The next step might be interpreting those coefficients. It is best to convert them to probabilities. First, I report the complete regression equation:
$y = -1.82 \times day1 - 2.92 \times day2 -2.78 \times day3 + 0.43 \times cohort2 + 0.31 \times cohort3 + 0.09 \times day1 \times groupB + 0.18 \times day2 \times groupB - 0.03 \times day3 \times groupB$, where the variables are indicators for the day and group.
So what is the $y$ for group A on day 1, cohort 1? We can use the regression equation. That would be -1.82. The same value for group B would be $-1.82 + 0.09=-1.73$. Next, we convert these values to probabilities using the formula, $(1+e^{-y})^{-1}$. We arrive at predicted probabilities of about 14% and 15% respectively.
Again, what is the $y$ for group A on day 2, cohort 3? We can use the regression equation. That would be $-2.92+0.31=-2.61$. The same value for group B would be $-2.92+0.31+0.18=-2.43$. Next, we convert these values to probabilities using the formula, $(1+e^{-y})^{-1}$. We arrive at predicted probabilities of about 6.8% and 8.1% respectively.
We can take the differences to arrive at the difference in probabilities between both groups on each of these days within cohorts. The statistical tests from earlier already tell us which days the differences in group probability of success are statistically different. We also know that overall, these group differences are worth considering.
When making predictions, you should probably not round your coefficients to two decimals places first, as you introduce error due to rounding. Use the full 8/9 digit coefficient reported by software to predict y, then transform to a probability, which you can then report to two decimal places if you want to.
