We are performing an A/B test involving web site activity, where samples in both our control and variant groups arrive gradually, day by day. Before we began our experiment, we did a power analysis on our population and came up with the sample size we would need for 80% power and standard 5% alpha to detect the effect size we were looking for. This certainly helped us plan the experiment.
However, we are about 3/4 of the way to collecting the specified sample size, but hypothesis tests are already showing a statistically significant effect. Since one purpose of power analysis is to determine the sample size that gives you an x% chance of detecting an effect if it is there (i.e. the chance to avoid a Type II error), does obtaining that specific sample count still matter once the effect is detected? At this point, we would be more worried about a Type I error rather than a Type II error.
I guess this is a practical question, but also sort of a larger question about an understanding of what power is really for. Yes, I've seen multiple warnings about "peeking" (e.g. here and here), but I don't quite understand how they relate power analysis to the Type I errors they warn about. Any insights appreciated - thanks.