According to this article, there's an issue with concluding a test as soon as it reaches significance (the "peeking effect"). Instead you need to determine the sample size ahead of time, and only look at the results once the test has gathered the predetermined number of observations.
This article also says that "statistical significance is not a stopping rule", and that just because a variation has 99% probability of beating the original it doesn't mean you can stop early. But what I don't understand fully is "why". If something has 99% probability of beating the original, that means given the observed effect you should make the correct decision 99% of the time?
So I have two questions:
Doesn't a confidence interval/significance account for the sample size? That is, even if the test swings wildly early on, an early 99% confidence still means that given the few observations, the results are still extreme enough to fall outside of the 99% of expected results? Isn't that the whole point of significance?
Let's say I do determine that I need to run each variation say 10K times each. At the end of this I find there is no conclusive evidence. So I decide to let the test continue to run for another 10K observations per variation. How does this affect power and significance?