I am new to AB testing/marketing data science. But would like to understand how to approach this problem i have. I have a model which predicts which customers are likely to be Families. We want to identify which are 'families' so that we can offer a product specific for families.
The model predicts 20k users to be families. We want to then send an email to these 20k users. But we also want to do this by probability threshold so we can validate the model e.g.: cohorts:
0.5-0.7 = 10k
0.7-0.9= 7k
0.9+ = 3k
The idea is that you send an email out to all of these customers in each then measure the uptake of the family product - IF model works you would expect higher conversion rate into family product at higher prob thresholds.
However we also want to test different marketing strategies, e.g. we want to have 3 tests:
1. Change text
2. Add colour highlighted box around some text
3. Add new font
So within each cohort above we would have the 3 tests in addition to control which is to send nothing. What i don't understand is, how do you choose a suitable sample size; and what is a suitable sample size to give a significant result?
e.g. if i was to split the people in 0.5-0.7 into 4 group ( 3 tests + control): i would have 2.5k in each of the groups to test each of the three test + control.
How do i apply a power calculation to see what best sample size is for each test within each cohort and is there anything flawed/ i have missed in above layout experiment?