Estimating comparative success of different brochures The Real World Problem
One of my clients is preparing to send a direct mailer to their subscribed user list, and this statistical challenge came up.
Their marketing team has 3 different brochures, and wants to know which brochure gets the highest response rate.  They would also like to know if sending the mailer with a hand-written address, on a thick envelope, improves results compared to a normal envelope.
Let's assume the following:


*

*For each brochure $b_i$ ($i = 1,2,3$), a person receiving that brochure who actually opens it and reads it will respond with probability $r_i$, where $r_i$ is the true response rate for that brochure

*The thick, high-quality envelopes have a true open rate of $o_{thick}$ while normal envelopes have an open rate of $o_{normal}$

*From previous mailings, we expect actual observed response rates will be between approximately 1% and 5%.  


Our Goals
We want to find the best brochure while sending the fewest number of mailers.  We also want to estimate the two open rates.
Upon gathering empirical response rates from actual sent mailers, if the true difference between response rates $r_i$ is greater than half a percent, we should be able to detect that difference as statisitcally significant with $p < .05$
My thoughts so far
We randomly assign users to each of the 3 brochures, such that $N$ users receive each brochure.  We want to know what $N$ we need to achieve our desired sensitivity in detecting differences in response rates.  Assuming the worst case, we need to be able to detect a difference between true rates of 1% and 1.5%.  The SD for this difference is $\sqrt{\frac{(.01*.99) + (.015*.985)}{N}}$.  Setting twice that quantity (2 standard deviations gives us 95% confidence) equal to .005 (our desired half a percent) leads to the solution $N = 3948$.
Questions


*

*Is this the optimal design or can we do better?

*Is my calculation of $N$ correct?


Finally, what is the best way to estimate $o_{normal}$ and $o_{thick}$, or simply the difference between the two?  
My idea was to randomly assign half of each brochure group to each type of envelope.  Within each brochure group, the observed response rates would be the product of the open rates and the $r_i$.  This would complicate my calculation of $N$ above, since really I should have used this product in my calculation.  
My answer would then depend on an estimate of the average open rate -- $\frac{o_{normal} + o_{thick}}{2}$ -- which I'd have to guess at.  Also, I am not sure how to determine the distribution of the difference between $o_{normal}$ and $o_{thick}$, since we now have three different estimates of that difference, each of which depends on a different $r_i$, each of which we have only empirical estimates of, empirical estimates which themselves depend on our guess at the average open rate.
Thanks very much for any help with this.
 A: There are empirical formulas for determining the sample size. The underlying test is a two-sample t test for equality of the metric( response rate in your case). Assuming that you want the power of the test to be 80%, one such formula is $n= 16\sigma^2/\Delta^2$ where $\sigma$ is the std dev of the metric ( response rate) and $\Delta$ is the amount of change in the response rate that you want to resolve reliably ( with statistical significance).
Also, there are fractional factorial designs available which let you optimize the number of trials (assuming you don't want to measure interactions of each factor with every other factor). This is a  survey paper on experimental design that describes the details.
A: Suppose that you sent brochures $A$ and $B$ to equal number of customers, then $a$ users respond to brochure $A$, and $b$ users respond to brochure $B$, and $b>a$.  Then the significance is
$P = {\sum_{n=b}^{a+b} C^{a+b}_n \over 2^{a+b}}$
It doesn't matter how many users received your brochures, just how many responded.
