A/B Test - Reset data measurements when changing audience size?

Question

I'm running an A/B test that originally was exposed to 10% of my traffic (5% variant / 5% control); The test has performed well thus far, and I'm looking to expand the size of the traffic pool to 50% (25% variant / 25% control).

Users are assigned to the [variant] / [control] / [not in experiment] buckets via a cookie hash.

My question is - Would there be a bias issue with combining the data collected from when the audience size was 10% with the data that will be collected when the audience size is 50%? Is it better to simply reset the measurements going forward?

One concern I have is that users that saw the experiment at 10% are now mixed with users who haven't seen it at 50%. However, this effect should be evenly distributed across both the control and variant.

Would love to hear some insights on this matter. Thanks!

Answer 1

I believe you should reset the data measurements. After changing the audience size, you will be over/under sampling with respect to time.

For example, suppose the website has different types of weekend and weekday traffic and each day there are 100 users. The metric on control is 60 on the weekdays and 55 on the weekends. The variant is 65 on the weekdays and 50 on the weekends. Just for simplicity, there is no other variance in the data.

control <- rep(c(60, 60, 60, 60, 60, 55, 55), 100)
variant <- rep(c(65, 65, 65, 65, 65, 50, 50), 100)
t.test(control, variant)

This gives a 95% confidence interval of [-2.673129, -1.612585].

Now, lets suppose the audience size was increased 5 times just in time for the weekend.

control <- c(rep(c(60, 60, 60, 60, 60), 100), rep(c(55, 55), 500))
variant <- c(rep(c(65, 65, 65, 65, 65), 100), rep(c(50, 50), 500))
t.test(control, variant)

Now the confidence interval is [1.289096, 2.044238]

In general, the users will always shift over time and cause problems with the test. Changing the audience size can increase those problems. And given that you'll achieve the same sample size 5 times faster, I would reset the data measurements.