# How do I perform a statistical test for the difference between of the change of A/B test?

I am performing an A(control) vs B(variant) test twice. I want to know if the difference of A/B change between the 2 runs is statistically significant.

Example:

1st run:
both A and B have 100 data points each
conversion(A)=12% , conversion(B)=15% => change=25%

2nd run:
both A and B have 80 data points each
conversion(A)=10%, conversion(B)=12% => change=20%



What test to apply to know if the difference between 25% and 20% is significant?

• Do you have the raw data? Counts? Aug 7, 2019 at 13:12
• This sounds almost like a difference in difference design. Aug 7, 2019 at 13:14
• I have only the data provided in the example. Aug 7, 2019 at 13:25

Assume that the conversion in groups A and B is binomially distributed with parameters $$p_1$$ and $$p_2$$, respectively. If you have the exact data, but not only the counts i.e. whether each datapoint belongs to A or not and to B or not, then I could suggest an approach which is probably not the best one, but it might be of some use:
• Take an estimates for $$p_1 = 0.12$$ and $$p_2 = 0.15$$, coming from run 1, which gives also an estimate for the difference $$p_2 - p_1 = 0.03$$. Then, use the Fisher's exact test(https://www.statsdirect.com/help/proportions/unpaired.htm), to get a confidence interval for the same difference , when using the data from run 2. If the obtained interval does not contain 0.03, then you might say that the difference between the 2 differences you have, for the proportions in groups A and B is significant.