I'm working on a problem/assignment where I'm given some activity logs from a website where an experiment was conducted. I'm asked to determine the earliest time when a statistically significant difference can be observed. Due to my background in stats, I'm puzzled with the whole idea of stopping an experiment when significance is observed, because my inclination would be to fix the sample beforehand and just wait for the experiment to conclude. I also came across this blog, which re-enforced my skepticism towards early stopping. Regardless, I need to answer the question I was given for this assignment. So, my question now becomes: how often do you test significance? Is is completely arbitrary, a function of time (i.e. every hour), a function of n (i.e. every 1000 new observations),...?
As the blog you link to mentions, repeated testing does impact significance. Because significance is based on the assumed sampling distribution under the null hypothesis, the sample space changes every time you "peek" at the data. That blog hints your options:
Until sequential or Bayesian experiment designs are implemented in software, anyone running web experiments should only run experiments where the sample size has been fixed in advance, and stick to that sample size with near-religious discipline.
- In frequentist testing, you may plan multiple tests—as many tests as you please, I suppose—provided you do the work of accurately assessing the null distribution. The number of checks is a function of $n$.
- Still frequentist, you could consider the size of the effect you'd like to detect, conduct a power analysis to determine an appropriate sample size, and conduct an experiment of that sample size. The number of checks is 1.
- Adopt a Bayesian approach, as detailed in a subsequent post by the same author. This would allow analysis at planned, or unplanned, times. (There may be a frequentist method of planning timed tests, but if so I'm unaware.)
There's a reason I call it work in (1). For example, say you intend to sample $n=1000$ visitors. Your sample space is all samples of $n = 1000$, and calculating a p-value is straightforward.
But suppose you intend to sample at most 1000, and intend to check significance at every interval of 100 visitors. A simple, straight-forward p-value for a sample of $n=200$ will not do. You didn't intend a sample of $n=200$; you intended a sequence of samples. This intention alters your sample space, and you must alter your significance calculations accordingly. Paid tools and the R package gsDesign can do this for analysis at planned stages.
The Bayesian approach in (3) would accept the posterior as the best possible inference given data and prior. This doesn't depend on a fixed sample size: it assumes only that observations are independent. This different approach sidesteps many of the concerns around using p-values.
However, it does require choosing a distribution that generates the data. In the case of binary data—click-through or not, for example—Bernoulli-generated data doesn't seem a problematic assumption. In other cases, the assumed distribution can deserve more scrutiny.