How to compare conversion rates from AB test when users visit multiple times [duplicate]

Question

I've been asked to compute the minimum sample size for an AB test that my company is conducting for a tweaked recommender system.

• We will choose two groups of users, one of which sees only the A version and one of which sees only the B version of their recommendations.
• We will compute conversion rate as the number of times users choose items from their recommendations divided by the number of times they view their recommendations.
• Many users won't view their recommendations at all during the test period while some users will view their recommendations dozens of times.

Question:

Should we calculate the overall conversion rate for each group and compare using a two-proportion Z-test?

Or should we calculate the conversion rate for each user who viewed their recommendations at least once and compare the average of the individual conversion rates for each group using a two-sample t-test?

I'm leaning toward the second option because it weights each user equally. Also, the first option would involve many observations that are not independent.

Answer 1

Both of your proposed methods may introduce some sort of bias to the result as the randomisation unit (a user) is not the same as the analysis unit (a recommendation view). As you pointed out, there is a dependency between users and views as some users will view many recommendations and others will view none, which leads us to a case of cluster randomisation. This affects the variance estimate, which in turn affects your t-statistic and p-value.

The common mitigation strategy in online experimentation is to use the delta method (Section 3 in [1]) or some kind of bootstrap (Section 2 in [2]). Chapters 14 and 18 of Kohavi et al.'s book on trustworthy online controlled experiments [3] give an accessible introduction to these techniques.

Another way, which will change your hypothesis, is to bring the analysis unit to the same level as randomisation unit. An example is to compute conversion rate on the user level, i.e. the number of users who has chosen an item from their recommendation at least once divided by the total number of users in the group.

Of course, this proposal will be useless if you are interested in whether your users will choose more items from your recommendations rather than having more users choose from your recommendations.

---

[1] Deng, Knoblich, and Lu (2018) Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas. In: KDD'18. https://arxiv.org/pdf/1803.06336.pdf
[2] Bakshy and Eckles (2013) Uncertainty in Online Experiments with Dependent Data: An Evaluation of Bootstrap Methods. In: KDD'13. https://arxiv.org/pdf/1304.7406.pdf
[3] Kohavi, Tang and Xu (2020) Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.