0
$\begingroup$

My motivation is to do A/A analysis using the "Bootstrap" method to calculate the confidence interval of the "sample mean" (of treatment and control) of all metrics.

In my A/B testing system, "users" (and not "events") are equally divided between treatment & control. A user can log multiple "events".

I understand that bootstrapping requires "sampling with replacement". In the A/A case, since the same user cannot be in both treatment and control at the same time,

  1. Should I randomly sample users to either treatment or control?
    • My concern is, since the same user can't be in both treatment and control, I might not be "sampling with replacement". Is this a true assumption?
  2. Or should I sample randomly from the events (with replacement)?
    • My concern here is, the same user's events can fall under both treatment and control for the same experiment which is generally not allowed in A/B tests.

What do you think is the best way to do this?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.