My motivation is to do A/A analysis using the "Bootstrap" method to calculate the confidence interval of the "sample mean" (of treatment and control) of all metrics.
In my A/B testing system, "users" (and not "events") are equally divided between treatment & control. A user can log multiple "events".
I understand that bootstrapping requires "sampling with replacement". In the A/A case, since the same user cannot be in both treatment and control at the same time,
- Should I randomly sample users to either treatment or control?
- My concern is, since the same user can't be in both treatment and control, I might not be "sampling with replacement". Is this a true assumption?
- Or should I sample randomly from the events (with replacement)?
- My concern here is, the same user's events can fall under both treatment and control for the same experiment which is generally not allowed in A/B tests.
What do you think is the best way to do this?