I studied statistics years ago and have forgotten it all so these may seem like general conceptual questions than anything specific but here is my issue.
I work for an e-commerce website as a UX Designer. We have an A/B testing framework that was built years ago which I am beginning to doubt it.
The metric we make all our decisions on is known as conversion, and it is based on the percentage of users that visit the site, end up purchasing something.
So we want to test changing the colour of the Buy button from Green to Blue.
The control is what we already have, the Green button where we know what our average conversion rate is. The experiment is replacing the Green button with the Blue button.
We agree 95% significance is the confidence level we are happy with and we turn the experiment on, leave it running.
When users visit the site, behind the scenes there is a 50/50 chance they will be sent to the control version (green button) Vs the experiment version (blue button).
After looking at the experiment after 7 days, I see a 10.2% increase in conversion in favour of the experiment with a sample size of 3000 (1500 going to the control, 1500 to the experiment) and a statistical significance of 99.2%. Excellent I think.
The experiment continues, the sample size grows and then I see a +9% increase in conversion with a significance of 98.1%. Ok, keep the experiment running longer and now the experiment shows just a 5% lift in conversion with a statistical significance of only 92%, with the framework telling me I need 4600 more samples before I reach 95% significance?
At what point is the experiment conclusive then?
If I think of say a clinical trial process where you agree on the sample size in advance and on completing the experiment you see a 10% improvement of whatever metric to 99% significance, then the decision is made that that drug then goes to market. But then if they’d done the experiment on 4000 people and they see a 5% improvement of whatever metric to only 92% significant then that drug wouldn’t be allowed to go to market.
Should we agree on a sample size in advance and stop once that sample size is reached and be happy with the results if the significance was 99% at the point of turning the experiment off?