If I run an A/B with the recommended sample size (using this for example), and at the end my results are not significant, what can be done at this point? If someone requests to continue running the test (i.e. increase the sample size), what implications does this have?
I guess this will cause a "peeking" problem, but how do you quantify this if someone asks you to, say, double the sample size? And if I do double the sample size, what is the likeihood of getting significant results, given that the recommended sample size did not produce significant results?
So a specific example
- I run a test with a recommended 10K impressions on each variation
- At the end of this test, the p-value is above 5% so there is not significant results
- Someone suggests to continue running the test to see if it does produce significant results. Is this a bad or good idea? And how do you motivate the answer?