So we introduced a new feature in our app, that would aid conversion (hypothetically). When I tried to measure this incremental change in conversion, I split my base set of customers into control (C) 30% and test (T) 70% sets via random sampling. Though feature was live to all customers for more than 2 months, we started an experiment say from Aug 19th (t1) where showed the feature only to the customers in test and control was not shown this feature.
What I observed is, for more than 1 month time period before t1 conversion of C was say 29.5%. And that of T's was 29% (both are averaged conversion calculated over 1 month before experiment start).
Now after Aug 19th (after t1) C's conversion was say 31% and T's conversion was say 31.2% (measured over 1-2 weeks after exp started).
Now, I want to calculate the improvement in conversion as follows:
Before t1: Diff in conversion (test-control) = 29-29.5 = -0.5%
After t1: Diff in conversion (test-control) = 31.2-31 = 0.2%
So change in conversion was 0.2-(-0.5) = 0.7%
So test added to an improved 0.7% conversion
Is this the right way to calculate when control set was already having a biased better performance than experiment set before start of experiment? (Assume i will run this exp for few more weeks & also do significance test further, but assume it was significant)
Some more info: The hoped improvement was also in the range of 1% itself. Plus, # of times control doing better than test before experiment start(t1) was about 67% now its come down to about 40%. Hence I am inclined to strongly argue test is adding ~0.7% value.