Calculating Sample Size During A/B

Question

Have spent a good bit of time googling this and could use some thoughts.

If we started an experiment and did not properly set stopping conditions and are now seeing significant p-values that have held--how can this be confirmed as a reasonable stopping point?

Is it unreasonable to calculate what the sample size estimate would have been given the lift we're seeing had we targeted that lift as the minimum prior to starting? And then wait for that sample size regardless of p-values? Taking the control as the 'baseline conversion' and treatment conversion / control conversion - 1 as the 'minimum improvement needed'.

This effectively becomes a dynamic sample size requirement that changes with the lift we're seeing (i.e. if lift today is 20%, and tomorrow it's 15%--we'll then see we need a larger sample size to confirm that 15% lift than we would have for 20%).

Is this better than just waiting and watching the p-value stabilize? It sounded like for sequential a/b testing we also needed a pre-defined N (which we didn't have) per: https://www.evanmiller.org/sequential-ab-testing.html

Answer 1

If we started an experiment and did not properly set stopping conditions and are now seeing significant p-values that have held--how can this be confirmed as a reasonable stopping point?

I don't think it can. Stopping the experiment because you've observed a significant difference is not a valid way of performing a sequential trial.

Is it unreasonable to calculate what the sample size estimate would have been given the lift we're seeing had we targeted that lift as the minimum prior to starting?

So what I think this is similar to is something called a post hoc power calculation. Basically, run your experiment and then perform a power calculation using the observed effect size to justify your results (e.g. The results should be trusted because we were powered to detect them, see?).

This is, unfortunately, invalid as explained here and elsehwere. Since the sample size is related to power, I imagine you would suffer the same draw backs as you would a post hoc power calculation.

If you did not specify an end point, I think the best you could do is imagine you didn't see the results and compute the required sample size to detect a smallest meaningful effect. In essence, tell me at which point the effect becomes too small to care. Then, design your experiment to detect that. The effect is either smaller than you care about (in which case failure to reject the null is no big deal) or it is bigger and you will detect it because you will have enough power to do so.

The problem here is that you've sort of poisoned the well. There is no way to remove that bias from your mind, or your stake holders mind. You can do your best to rationalize a smallest meaningful effect using business logic, but the bias will remain.

Now, this is a strictly statistical answer. In practice, I doubt doing this for one experiment will lead to much harm, especially if it is for some sort of conversion and not for something that will impact/harm people. But running experiments like this repeatedly will surely result in an erroneous inference eventually.