I am working on an exercise using conversion rate data on a travel website. The conversion rate is defined as the number of users in a given time period that make a purchase.
There are two groups, A and B. I am trying to understand if a new feature released to group B had an impact on conversion rate. There are underlying differences between the two groups (causes unknown). I therefore wish to take a difference-in-differences approach using data prior to the feature release, and after the feature release.
The data is as follows:
I am familiar with using OLS regression to analyse difference-in-differences for a continuous metric (for example - revenue). However, I am looking to understand how I can do the same for a binomial metric.
- Can a t-test (or similar) be applied to the aggregate conversion rates to understand the significance (or lack thereof) of the difference-in-difference value. If so, how is 'sample size' defined given we have 4 groups but 2 differences to compare
- Can a logistic (or similar) regression model be applied to these aggregate values. In this case, how would the variance (and therefore P value), be calculated
Note that the parallel trends assumption has been validated using a daily breakdown of the data. However, due to the nature of conversion rates, the daily values are not equal to the aggregate values (for example - a customer could visit the website on multiple days but only convert once, hence the aggregated conversion rate will not equal the daily conversion rate).