# AB testing: Control was performing 0.5% better than experiment set before the initiation of experiment

So we introduced a new feature in our app, that would aid conversion (hypothetically). When I tried to measure this incremental change in conversion, I split my base set of customers into control (C) 30% and test (T) 70% sets via random sampling. Though feature was live to all customers for more than 2 months, we started an experiment say from Aug 19th (t1) where showed the feature only to the customers in test and control was not shown this feature.

What I observed is, for more than 1 month time period before t1 conversion of C was say 29.5%. And that of T's was 29% (both are averaged conversion calculated over 1 month before experiment start).

Now after Aug 19th (after t1) C's conversion was say 31% and T's conversion was say 31.2% (measured over 1-2 weeks after exp started).

Now, I want to calculate the improvement in conversion as follows:

Before t1: Diff in conversion (test-control) = 29-29.5 = -0.5%

After t1: Diff in conversion (test-control) = 31.2-31 = 0.2%

So change in conversion was 0.2-(-0.5) = 0.7%

So test added to an improved 0.7% conversion

Is this the right way to calculate when control set was already having a biased better performance than experiment set before start of experiment? (Assume i will run this exp for few more weeks & also do significance test further, but assume it was significant)

Some more info: The hoped improvement was also in the range of 1% itself. Plus, # of times control doing better than test before experiment start(t1) was about 67% now its come down to about 40%. Hence I am inclined to strongly argue test is adding ~0.7% value.

• You don't need a difference in difference. You randomly assigned users to treatment and control, so their baseline difference is 0 in expectation. All you need to do is compare post assignment conversion rates. Commented Sep 2, 2023 at 12:51
• You just discovered that there is random variation in your two samples. That's fine and that being said, that said, DiD is fine to use. Using a quasi-experimental methodology with RCT/AB data doesn't invalidate the findings. Commented Sep 2, 2023 at 17:29

Prior to $$t_1$$, your users are part of the same population, so the distribution of their future outcomes is the same. Hence, the conversion rate for both treatment and control prior to $$t_1$$ is the same, meaning the null is true. No need to estimate the difference; we know its 0.