Determining sample size before starting an experiment or run the experiment indefinitely?

Question

I studied statistics years ago and have forgotten it all so these may seem like general conceptual questions than anything specific but here is my issue.

I work for an e-commerce website as a UX Designer. We have an A/B testing framework that was built years ago which I am beginning to doubt it.

The metric we make all our decisions on is known as conversion, and it is based on the percentage of users that visit the site, end up purchasing something.

So we want to test changing the colour of the Buy button from Green to Blue.

The control is what we already have, the Green button where we know what our average conversion rate is. The experiment is replacing the Green button with the Blue button.

We agree 95% significance is the confidence level we are happy with and we turn the experiment on, leave it running.

When users visit the site, behind the scenes there is a 50/50 chance they will be sent to the control version (green button) Vs the experiment version (blue button).

After looking at the experiment after 7 days, I see a 10.2% increase in conversion in favour of the experiment with a sample size of 3000 (1500 going to the control, 1500 to the experiment) and a statistical significance of 99.2%. Excellent I think.

The experiment continues, the sample size grows and then I see a +9% increase in conversion with a significance of 98.1%. Ok, keep the experiment running longer and now the experiment shows just a 5% lift in conversion with a statistical significance of only 92%, with the framework telling me I need 4600 more samples before I reach 95% significance?

At what point is the experiment conclusive then?

If I think of say a clinical trial process where you agree on the sample size in advance and on completing the experiment you see a 10% improvement of whatever metric to 99% significance, then the decision is made that that drug then goes to market. But then if they’d done the experiment on 4000 people and they see a 5% improvement of whatever metric to only 92% significant then that drug wouldn’t be allowed to go to market.

Should we agree on a sample size in advance and stop once that sample size is reached and be happy with the results if the significance was 99% at the point of turning the experiment off?

Answer 1

I think the concept you are searching for is sequential analysis. There are a number of questions on this site tagged with the term which you might find useful, perhaps Adjusting the p-value for adaptive sequential analysis (for chi square test)? would be a place to start. You could also consult the Wikipedia article here. Another useful search term is alpha spending which comes from the fact that as you take each repeated look you should regard it as using up some of your alpha (significance level). If you keep peeking at your data without taking the multiple comparisons into account you run into the sort of problem which you outline in your question.

Answer 2

At what point is the experiment conclusive then?

I think this is where the error in thinking is. There is no point at which the experiment can be "conclusive" if you take that to mean "deductively prove causation". When you're doing an experiment that involves a statistical test, you need to make a commitment regarding what evidence you consider to be good enough.

Statistically-sound experimental procedures give you results with known rates of false positives and false negatives. If you have chosen a procedure that uses 0.05 as the threshold for significance, you're saying that you are willing to accept that in 5% of the cases where there is actually no difference, your test will tell you that there is a difference.

If you deviate from the procedure in the ways you describe (not choosing a stopping point ahead-of-time, simply running the test until your computed p-value drops below 0.05, or running the entire experiment multiple times until you get a positive result, etc.), you are making it more likely that your test will tell you that a difference exists when there is in fact no difference. You are making it more likely that you will get fooled into thinking your change has been effective. Don't let yourself get duped.

Read this paper: False-Positive Psychology Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant

It highlights several ways that you can improperly interfere with a testing procedure that make it more likely for you to get fooled, including the exact scenario you describe (not knowing when to stop an experiment).

Other answers give you some solutions to mitigate against these problems (sequential analysis, Bonferroni correction for multiple comparisons). But those solutions, while able to control the false-positive rate, typically reduce the power of the experiment, making it less likely to detect differences when they do exist.

---

There is one other error you're making. You talk about a "10% improvement of whatever metric to 99% significance". Significance tests only can tell you whether the observed difference in your sample is likely to be due to a real underlying difference or just random noise; they do not give you confidence intervals around what the true magnitude of the difference is.

Answer 3

I think you are asking the wrong question here. The question you are asking is about statistical tests; I think the right question is "why is the effect changing over time?"

If you are measuring a 0/1 variable for conversion (did they buy at all?) then people who did not buy in an initial session may come back and buy later. This means that the conversion rate will increase over time and any effect of having a customer purchase in their first visit as opposed to later visits will be lost.

In other words, first get right what you are measuring, then worry about how you are measuring.

Answer 4

This is exactly why a clear criterion needs to be defined ahead of trials. As @mdewey indicates there are established methods for periodically evaluating a trial but these all require a clear stopping criteron to prevent any fudging over the decision. Two critical issues are that you need to correct for multiple comparisons and that each analysis is not independent, but its outcome is heavily influenced by the results of your previous analyses.

As an alternative it may be best practice to define a set sample size based on commercially relevant arguments.

First the company should agree what a commercially relevant change in conversion rate is (i.e. what size of difference is needed to warrant making a commercial case for the change to be deployed permanently). Without agreeing this there is no sensible benchmark.

Once the minimum commercially relevant effect size is determined (note this may change on a case by case basis depending on how critical the step being tested is) then you agree the level of risk that the company is willing to accept for missing a true effect (beta) and for accepting a false effect (alpha).

Once you have these numbers plug them into the sample size calculator and voila, you will have your set sample size to make a decision.

---

EDIT

Using small sample sizes and hoping they will show big enough effect is a false economy (since your aim is actionable dependable results rather than generating controversial hypothesis for academic publication). Assuming unbiased sampling, at low sample sizes the probability of randomly selecting samples that happen to be all towards opposite extremes is higher than in high sample sizes. This leads to a higher likelihood of rejecting a null hypothesis when in fact there is no difference. So this would mean pushing through changes that are not actually making a real impact or even worse having a slightly negative impact. This is a different way of explain what @Science is talking about when they state

"you are making it more likely that your test will tell you that a difference exists when there is in fact no difference"

The point of pre-specifying your statistical analysis (whether a fixed sample size as I describe or a multiple evaluation strategy) is that you appropriately balance the demands of both type I and II errors. Your current strategy appears to focus on type I errors and completely ignore type II.

As numerous other answerers have stated the results never are conclusive, but if you have considered both type I and II errors and their impact on your business then you will have the most confidence you can hope for whether to implement changes based on the results. In the end decision making is about being comfortable with your level of risk and never treat your 'facts' as immutable.

I am intrigued by other aspects of your study design that may be influencing the results you see. They may be revealing some subtle factors that aren't what you want.

Are the people selected for the sample all new visitors, all returning visitors or is that undifferentiated? Established customers may have an increased tendency to go for something novel (so biased towards change not a specific color), but to new customers everything is new.

Do the actual people clicking recurr within the timeframe of the study?

If people visit multiple times over the timeframe of the study do they get presented with the same version or is it randomly allocated on the fly?

If recurring visitor are included there a danger of exposure fatigue (it is no longer distracting because it is no longer new)

Answer 5

Common practice usually dictates that you decide on the sample size first (to control the statistical power of your hypothesis test), and then perform the experiment.

In response to your current position, it sounds like you're after combining a series of hypothesis tests. I recommend you look at Fisher's method. In addition, you're probably going to want to look at Brown's or Kost's methods for accommodating Fisher's method to dependent test statistics. As another respondent has mentioned, a customer's conversion (or non-conversion) is going to impact whether they will make a purchase (or not) on the next visit - regardless of what color the button is.

Afterthoughts:

1. More information and sources on Fisher's methods and their extensions can be found on the Wikipedia article for Fisher's method.
2. I feel it is important to mention that an experiment is never really conclusive. A small p-value does not indicate that your result is conclusive - only that the null hypothesis is unlikely based on the data you've acquired.