In AB test do the sample size calculations represent the minimum or maximum size required for measuring difference between the test and control

Question

A quick question on the frequentest approach of sample size calculation.

I recently came across this below note:

The sample size required should be strictly enforced; if the power analysis shows that each test variant requires 100,000 users, the test should only be evaluated based on the first 100,000 users that are bucketed. If this is not strictly adhered to, we expose ourselves to uncertainty

I was under the assumption that the sample size calculation provides the minimum sample size that is required to measure the difference between the treatment and control and is related to the minimum detectable effect but having more data would not necessarily be a problem.

Answer 1

When you do a sample size calculation for a desired power level, the result is the number of observations required to achieve that power. Any more than that and your power increases; any fewer and your power decreases.

What I think that passage is saying is that there should be no reason to interfere with assignment to variant (e.g. you should not prevent someone from getting some particular variant because of the behavior they display on your platform). Doing so once isn't so bad, but if it is a systematic thing, you might mess up the causal interpretation of the test you perform. It is basically saying to be honest and not fiddle with your experiment too much.