Difference-in-differences - Binary data

I am working on an exercise using conversion rate data on a travel website. The conversion rate is defined as the number of users in a given time period that make a purchase.

There are two groups, A and B. I am trying to understand if a new feature released to group B had an impact on conversion rate. There are underlying differences between the two groups (causes unknown). I therefore wish to take a difference-in-differences approach using data prior to the feature release, and after the feature release.

The data is as follows:

testGroup timeGroup users conversions cvr
A pre-test 18350 550 0.036
A test 19441 782 0.040
B pre-test 19824 835 0.042
B test 18963 946 0.050

I am familiar with using OLS regression to analyse difference-in-differences for a continuous metric (for example - revenue). However, I am looking to understand how I can do the same for a binomial metric.

Specifically:

• Can a t-test (or similar) be applied to the aggregate conversion rates to understand the significance (or lack thereof) of the difference-in-difference value. If so, how is 'sample size' defined given we have 4 groups but 2 differences to compare
• Can a logistic (or similar) regression model be applied to these aggregate values. In this case, how would the variance (and therefore P value), be calculated

Note that the parallel trends assumption has been validated using a daily breakdown of the data. However, due to the nature of conversion rates, the daily values are not equal to the aggregate values (for example - a customer could visit the website on multiple days but only convert once, hence the aggregated conversion rate will not equal the daily conversion rate).

Often in a 'difference-in-differences' design the differences in the A group are paired and also the differences in the B group are paired. I see no pairing here. The pre and post results in A may be from unrelated randomly chosen subjects and similarly from B. [BTW, your conversions rate for A-pre seems incorrect: $$550/18\,350 \approx 0.030].$$

So this is not a standard D in D design. I will show one possible way to look at your data, but I will not be surprised if others have alternative suggestions.

In A, treating the pre and post proportions (0.03 and 0.04) as if from independent 'conversions' in the R procedure prop.test we find them to be highly significantly different (P-value near 0) with a 95% confidence interval $$(0.0065, 0,0140)$$ for the difference.

prop.test(c(782, 550), c(19441, 18350), cor=F)

2-sample test for equality of proportions
without continuity correction

data:  c(782, 550) out of c(19441, 18350)
X-squared = 29.175, df = 1, p-value = 6.613e-08
alternative hypothesis: two.sided
95 percent confidence interval:
0.006548142 0.013954890
sample estimates:
prop 1     prop 2
0.04022427 0.02997275

Similarly, in B the two proportions (0.042 and 0.050) are highly significantly different with CI $$(0.0036,0.0199)$$ for the difference.

2-sample test for equality of proportions
without continuity correction

data:  c(946, 835) out of c(18963, 19824)
X-squared = 13.343, df = 1, p-value = 0.0002595
alternative hypothesis: two.sided
95 percent confidence interval:
0.003592234 0.011939685
sample estimates:
prop 1     prop 2
0.04988662 0.04212066

Thus, the confidence intervals for the gains in A and B overlap. Conversion rates increased significantly from pre to post in both groups, but not by remarkably different amounts. (For example, one could argue that the true conversion rate in both groups may have increased by about 0.9%.)