I am running an A/B test. I have about 95k samples per variation and 1.29% relative improvement. The test is to see whether the variant converts users to booking better than the control. Test is still running. The test has been running for about 2 months. I cant afford to run it any longer. I am using Optimizely (fullstack) to run this test. So, I didnt have to calculate the sample size before the test and wait for it. Now, when I look at the Optimizely results dashboard , it shows as less than 1% statistical significance. However, if I plug the numbers into a chi-square test calculator like this (https://www.evanmiller.org/ab-testing/chi-squared.html#!31984/96684;32299/96391@95) , I get a statistically significant result (p value .0463). I understand that there is a multiple testing problem that can inflate type 1 error rate and I might be getting a false positive. How can I adjust for this and get an accurate result with the given sample size and relative improvement, so far ?


There are no easy p-value adjustments. Your real question is about whether you can stop your test, calculate a p-value, and rely on the answer. I can think of 3 paths forward. (1) You already said that you can't afford to run the test any longer. Stop the test. Calculate the p-value when you stop the test. Understand that there is risk of not following your test plan leading to a potentially invalid result. (2) Now that you have a better idea of the potential improvement and the rate of impressions, calculate a proper sample size to detect the improvement of interest, restart the test, and wait until you have reached the desired sample size or wait time. There are no short cuts. (3) Contact Optimizely and ask them to tell you how they do their calculation and how it is different from a chi-squared test. You bought the product, they should be happy to help.

  • $\begingroup$ There is also the possibility sequential-analysis $\endgroup$ Jan 11 '21 at 1:50
  • 2
    $\begingroup$ On 3), Optimizely uses a variant of sequential test called Mixed Sequential Probability Ratio test, which provides a valid p-value at any point in time. $\endgroup$
    – B.Liu
    Jan 11 '21 at 1:50
  • $\begingroup$ I agree that it is a possibility. However, I am very strict in that I think you need to specify the type of statistical test you are going to use before the experiment. We don't know if a researcher switches to a sequential test mid-experiment is doing it for improper reasons or has been biased by looking at the results (like this one). There really are no short-cuts once the experiment has started. Stick to the test plan. $\endgroup$
    – R Carnell
    Jan 11 '21 at 1:53
  • $\begingroup$ @RCarnell, thank you for the quick response. I did option #2 in your response. So, the required sample to detect a 1.29% improvement (with 80% power & .05 aplha) is about 190k samples per variation. So, it makes sense that Optimizely is showing less than 1% stat sig. But, it's hard to explain this to other team members when they plug in the number in such online significance calculators and see a significant result. I was trying to find a way to show them that we need to apply a certain penalty/adjustment to the p-value to account for the peeking. $\endgroup$
    – bp0308
    Jan 11 '21 at 2:29
  • $\begingroup$ Also, this leads to another question about statistical tests in general (outside of A/B test) for any historical dataset. For example, if I have a dataset from past year that I can apply a Chi-square test to compare two groups in the dataset. How reliable is the result in this case? Would the result have changed If more data was added to the dataset? is this case similar to peeking in an ab test? if no, why? $\endgroup$
    – bp0308
    Jan 11 '21 at 2:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.