I am running an A/B test. I have about 95k samples per variation and 1.29% relative improvement. The test is to see whether the variant converts users to booking better than the control. Test is still running. The test has been running for about 2 months. I cant afford to run it any longer. I am using Optimizely (fullstack) to run this test. So, I didnt have to calculate the sample size before the test and wait for it. Now, when I look at the Optimizely results dashboard , it shows as less than 1% statistical significance. However, if I plug the numbers into a chi-square test calculator like this (https://www.evanmiller.org/ab-testing/chi-squared.html#!31984/96684;32299/96391@95) , I get a statistically significant result (p value .0463). I understand that there is a multiple testing problem that can inflate type 1 error rate and I might be getting a false positive. How can I adjust for this and get an accurate result with the given sample size and relative improvement, so far ?
-
$\begingroup$ See towardsdatascience.com/… $\endgroup$– kjetil b halvorsen ♦Commented Jan 22, 2021 at 15:11
1 Answer
There are no easy p-value adjustments. Your real question is about whether you can stop your test, calculate a p-value, and rely on the answer. I can think of 3 paths forward. (1) You already said that you can't afford to run the test any longer. Stop the test. Calculate the p-value when you stop the test. Understand that there is risk of not following your test plan leading to a potentially invalid result. (2) Now that you have a better idea of the potential improvement and the rate of impressions, calculate a proper sample size to detect the improvement of interest, restart the test, and wait until you have reached the desired sample size or wait time. There are no short cuts. (3) Contact Optimizely and ask them to tell you how they do their calculation and how it is different from a chi-squared test. You bought the product, they should be happy to help.
-
$\begingroup$ There is also the possibility sequential-analysis $\endgroup$ Commented Jan 11, 2021 at 1:50
-
3$\begingroup$ On 3), Optimizely uses a variant of sequential test called Mixed Sequential Probability Ratio test, which provides a valid p-value at any point in time. $\endgroup$– B.LiuCommented Jan 11, 2021 at 1:50
-
$\begingroup$ I agree that it is a possibility. However, I am very strict in that I think you need to specify the type of statistical test you are going to use before the experiment. We don't know if a researcher switches to a sequential test mid-experiment is doing it for improper reasons or has been biased by looking at the results (like this one). There really are no short-cuts once the experiment has started. Stick to the test plan. $\endgroup$ Commented Jan 11, 2021 at 1:53
-
$\begingroup$ @RCarnell, thank you for the quick response. I did option #2 in your response. So, the required sample to detect a 1.29% improvement (with 80% power & .05 aplha) is about 190k samples per variation. So, it makes sense that Optimizely is showing less than 1% stat sig. But, it's hard to explain this to other team members when they plug in the number in such online significance calculators and see a significant result. I was trying to find a way to show them that we need to apply a certain penalty/adjustment to the p-value to account for the peeking. $\endgroup$– bp0308Commented Jan 11, 2021 at 2:29
-
$\begingroup$ Also, this leads to another question about statistical tests in general (outside of A/B test) for any historical dataset. For example, if I have a dataset from past year that I can apply a Chi-square test to compare two groups in the dataset. How reliable is the result in this case? Would the result have changed If more data was added to the dataset? is this case similar to peeking in an ab test? if no, why? $\endgroup$– bp0308Commented Jan 11, 2021 at 2:51