2
$\begingroup$

Imagine you're doing an A/B-test in Marketing. You send a control group the standard e-mail, and a test group an e-mail in which one element was changed (for example the color of a button). You want to compare the conversion rates of the e-mails, to determine whether the new e-mail leads to a different conversion rate.

Suppose that the base conversion rate is 16%, and you're interested in a Minimun Detectable Effect of at least 2 percentage-points. Power is set at 80% and the significance level at 5%.

In order to determine the minimum number of recipients per version, I used this calculator:

https://www.evanmiller.org/ab-testing/sample-size.html#!16;80;5;2;0

The suggested sample size per variation is 5,352 recipients per version.

However, if I use the calculator below to determine whether a result is significant, I can find significance at much smaller sample sizes:

https://www.socscistatistics.com/tests/ztest/default2.aspx

For example, if I enter 0.16 and 0.18 (two percentage-points higher) as achieved proportions (conversion rates) and sample sizes of 2,725 for each sample, the result is already significant.

Question

Why would the first calculator suggest a much larger sample size than is required for significance, for the same resulting proportions? (or in more general terms: is it true that the sample size should be much larger than what seems to be minimally required to achieve significance? And if so, why?)

Own guess

My guess is that it has something to do with uncertainty and variability: a larger sample size would make it more likely to find a detectable effect in case of some random noise. I also thought it might have to do with that the first calculator considers a double-sided hypothesis (the effect can go either way). But then in the second calculator, I do select the option for two-tailed test. As you can see I'm not sure about it, and of course these explanations are not very formal and well-formulated. Hence any help would be greatly appreciated.

$\endgroup$
5
$\begingroup$

It's your power. Note in that first calculator link you can set the power downward to 60%, and that lowers the sample needed to 3310. If you could lower the power there to 50%, you would likely get 2725.

When you make that significance test using the second calculator, you are at a critical point. If you had one less positive result, you wouldn't be statistically significant. That's essentially coin flip, i.e. a 50-50 chance you'd find a difference. You want a better chance than that to find a 2% difference if it really exists (specifically, you asked for an 80% chance).

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.