My motivation is to do A/A analysis using the "Bootstrap" method to calculate the confidence interval of the "sample mean" (of treatment and control) of all metrics.

In my A/B testing system, "users" (and not "events") are equally divided between treatment & control. A user can log multiple "events".

I understand that bootstrapping requires "sampling with replacement". In the A/A case, since the same user cannot be in both treatment and control at the same time,

  1. Should I randomly sample users to either treatment or control?
    • My concern is, since the same user can't be in both treatment and control, I might not be "sampling with replacement". Is this a true assumption?
  2. Or should I sample randomly from the events (with replacement)?
    • My concern here is, the same user's events can fall under both treatment and control for the same experiment which is generally not allowed in A/B tests.

What do you think is the best way to do this?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.