What sample size is good enough for this AB Test experiment?

Question

I am new to AB testing/marketing data science. But would like to understand how to approach this problem i have. I have a model which predicts which customers are likely to be Families. We want to identify which are 'families' so that we can offer a product specific for families.

The model predicts 20k users to be families. We want to then send an email to these 20k users. But we also want to do this by probability threshold so we can validate the model e.g.: cohorts:

0.5-0.7 = 10k
0.7-0.9= 7k 
0.9+ = 3k

The idea is that you send an email out to all of these customers in each then measure the uptake of the family product - IF model works you would expect higher conversion rate into family product at higher prob thresholds.

However we also want to test different marketing strategies, e.g. we want to have 3 tests:

1. Change text
2. Add colour highlighted box around some text 
3. Add new font

So within each cohort above we would have the 3 tests in addition to control which is to send nothing. What i don't understand is, how do you choose a suitable sample size; and what is a suitable sample size to give a significant result?

e.g. if i was to split the people in 0.5-0.7 into 4 group ( 3 tests + control): i would have 2.5k in each of the groups to test each of the three test + control.

How do i apply a power calculation to see what best sample size is for each test within each cohort and is there anything flawed/ i have missed in above layout experiment?

Answer 1

I don't think you need to do a power calculation.

Power calculations are usually done prior to the experiment being conducted. We select a smallest meaningful effect we would like to detect, act as if that effect is the true effect size, and determine the number of samples we would need in order to reject the null with some probability (the power of the test).

You've already collected the samples (20K users) you need for your experiment, so a power calculation is moot. It could be the case you need fewer than 20K samples in order to achieve the desired power for your test (so as not to be wasteful), but I am willing to bet this is not an issue.