I am running an A/B test. I have about 95k samples per variation and 1.29% relative improvement. The test is to see whether the variant converts users to booking better than the control. Test is still running. The test has been running for about 2 months. I cant afford to run it any longer. I am using Optimizely (fullstack) to run this test. So, I didnt have to calculate the sample size before the test and wait for it. Now, when I look at the Optimizely results dashboard , it shows as less than 1% statistical significance. However, if I plug the numbers into a chi-square test calculator like this (https://www.evanmiller.org/ab-testing/chi-squared.html#!31984/96684;32299/96391@95) , I get a statistically significant result (p value .0463). I understand that there is a multiple testing problem that can inflate type 1 error rate and I might be getting a false positive. How can I adjust for this and get an accurate result with the given sample size and relative improvement, so far ?
There are no easy p-value adjustments. Your real question is about whether you can stop your test, calculate a p-value, and rely on the answer. I can think of 3 paths forward. (1) You already said that you can't afford to run the test any longer. Stop the test. Calculate the p-value when you stop the test. Understand that there is risk of not following your test plan leading to a potentially invalid result. (2) Now that you have a better idea of the potential improvement and the rate of impressions, calculate a proper sample size to detect the improvement of interest, restart the test, and wait until you have reached the desired sample size or wait time. There are no short cuts. (3) Contact Optimizely and ask them to tell you how they do their calculation and how it is different from a chi-squared test. You bought the product, they should be happy to help.