I am performing an A(control) vs B(variant) test twice. I want to know if the difference of A/B change between the 2 runs is statistically significant.


1st run: 
both A and B have 100 data points each
conversion(A)=12% , conversion(B)=15% => change=25%

2nd run: 
both A and B have 80 data points each
conversion(A)=10%, conversion(B)=12% => change=20%

What test to apply to know if the difference between 25% and 20% is significant?

  • $\begingroup$ Do you have the raw data? Counts? $\endgroup$ Aug 7, 2019 at 13:12
  • $\begingroup$ This sounds almost like a difference in difference design. $\endgroup$ Aug 7, 2019 at 13:14
  • $\begingroup$ I have only the data provided in the example. $\endgroup$
    – Anton
    Aug 7, 2019 at 13:25

1 Answer 1


Assume that the conversion in groups A and B is binomially distributed with parameters $p_1$ and $p_2$, respectively. If you have the exact data, but not only the counts i.e. whether each datapoint belongs to A or not and to B or not, then I could suggest an approach which is probably not the best one, but it might be of some use:

  • Take an estimates for $p_1 = 0.12$ and $p_2 = 0.15$, coming from run 1, which gives also an estimate for the difference $p_2 - p_1 = 0.03$. Then, use the Fisher's exact test(https://www.statsdirect.com/help/proportions/unpaired.htm), to get a confidence interval for the same difference , when using the data from run 2. If the obtained interval does not contain 0.03, then you might say that the difference between the 2 differences you have, for the proportions in groups A and B is significant.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.