A quick question on the frequentest approach of sample size calculation.

I recently came across this below note:

The sample size required should be strictly enforced; if the power analysis shows that each test variant requires 100,000 users, the test should only be evaluated based on the first 100,000 users that are bucketed. If this is not strictly adhered to, we expose ourselves to uncertainty

I was under the assumption that the sample size calculation provides the minimum sample size that is required to measure the difference between the treatment and control and is related to the minimum detectable effect but having more data would not necessarily be a problem.

  • $\begingroup$ I'm having trouble squaring the idea of "stopping rules" with the setting of this question, which appears to concern fixed sample sizes. Could you elaborate on this apparent inconsistency? What stopping rules are you referring to? $\endgroup$ – whuber Jan 14 '19 at 19:59
  • $\begingroup$ @whuber: Hi, i removed that line about "stopping rules". Reading the question again, i realized that's actually not related to my doubt. Regards. My question is "is the sample size the absolute maximum" or will it have an impact on the measurement if i have more data points than the calculated sample size. $\endgroup$ – Raj Jan 14 '19 at 20:10

When you do a sample size calculation for a desired power level, the result is the number of observations required to achieve that power. Any more than that and your power increases; any fewer and your power decreases.

What I think that passage is saying is that there should be no reason to interfere with assignment to variant (e.g. you should not prevent someone from getting some particular variant because of the behavior they display on your platform). Doing so once isn't so bad, but if it is a systematic thing, you might mess up the causal interpretation of the test you perform. It is basically saying to be honest and not fiddle with your experiment too much.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.