How to perform a comparison of means test in a scenario where a user can have multiple conversions? In the case where a user can have at most 1 conversion, one can simply do a comparison of proportions test. However, in a case where a user may have multiple conversions, I am unsure how to model this variable? A Bernoulli assumption is no longer correct.

As an example of what I am looking for: Suppose we have two variants A and B. A is the control and B contains a new feature. I would like to see if variant B leads to more conversions (given the same number of users). As each user can have multiple conversions, I cannot simply do t-test using the proportions of A and B. How would you handle this situation?

  • $\begingroup$ What do you want to achieve? I am not sure what "comparison of means" refers to. Do you want to know whether the difference is significant, or do you just want to measure the difference between A and B or do you want to use it in a causal effect study? $\endgroup$
    – frank
    Jun 23 at 12:58
  • $\begingroup$ Does this answer your question? A/B testing ratio of sums $\endgroup$
    – xiaoA
    Jun 26 at 10:08


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.