I am trying to design an email test to measure the demand lift obtained from a marketing promotion (treatment) versus no promotion (control). To do so, I want to calculate the per-group sample size required to get a significant read on the difference in average demand per-customer for different marketing segments.
To do so, I am applying the following formula (for each segment):
$$ N = \frac{2(Z_{1-\alpha/2}+Z_{\pi})^2\sigma^2}{\Delta^2} $$
Where:
$Z_{1-\alpha/2}$ = percentile of the normal distribution used as the critical value in a two-tailed test (1.96)
$Z_{\pi}$ = percentile of the normal distribution where $\pi$ is the power of the test (0.84 for 80th percentile)
$\sigma$ = within-group standard deviation
$\Delta$ = expected mean difference between the treatment versus control population
To calculate the standard deviation and expected mean difference above, I pulled historical response for the same period last year during which the test will run. My question is this: should the group means and standard deviations be estimated from the total population which was exposed to the treatment (and control), respectively, or should the mean and standard deviation be calculated based on respondents only? Put another way, should I use the mean/variance for the full audience exposed to a given treatment in the past, or the mean/variance for responders only, and then back solve for required full audience?
The results that I’m getting appear counter-intuitive, with similar required sample sizes among the most-engaged and least-engaged audiences, so I know I must be doing this wrong.
Most of the material that I’ve come across from the marketing community involves using a desired difference in response rate to solve for appropriate per-group sample sizes. In my case, however, the metric of interest is demand-based rather than raw response (average demand per customer). That said, the response rate is an important metric, as it is particularly low for certain groups of customers, but it does not directly reflect the metric of interest.
Thanks in advance!