What is the correct way to determine the amount of difference between two proportions?

Suppose I am testing the effectiveness of a new piece of machinery. The current machine has a success rate of 20%, that is, for every hundred widgets produced, 20 can be sold.

Now I've developed a new prototype and I want to know if it performs better. Additionally, it only makes sense to invest in the new machine if it is 10% better, i.e. a 22% success rate.

So I set up an experiment and collect this data:

machine     attempts   successes
---------------------------------
Original      n1         k1
Prototype     n2         k2


Assume that
* alpha= 5%
* beta= 20%
* I'd like to use the smallest sample size possible.

My understanding is that normally I'd want a 2-tailed difference of proportions test. But that test would only tell me if the two machines performed differently, not if one was 10% or more better.

How do I determine what sample size to use, what is the appropriate test(s) to use, and how do I report the magnitude of the difference?

-
 How large are $n_1$ and $n_2$? Tests for comparing proportions tend to lose power if the sample sizes differ too much. – MånsT Jul 20 '12 at 6:03 If the new machine has a $21.8\%$ success rate, you would like to reject it unless the original really only succeeds about $19.8\%$ of the time? Does testing the original cost the same as testing the prototype? – Douglas Zare Jul 20 '12 at 6:16 @MånsT n1 and n2 could be between 1,000 to 100,000 if necessary. I can keep them roughly equal if that helps power. – dan Jul 20 '12 at 13:46 @Douglas Zare testing both machines costs the same. The business case is that the cost of fully developing the prototype would only pay off if I can expect a 10% improvement. – dan Jul 20 '12 at 14:05 @dan: I have some recommendations, but I don't have the time to post them now. I'll write an answer on Sunday or Monday unless threre already is a thorough answer by then! – MånsT Jul 20 '12 at 20:17
show 1 more comment

The standard hypothesis test only tells you that you are significantly better but not by the magnitude. To show that the magnitude is greater than a specified $\Delta$ requires a larger sample size. Instead of testing $p_1-p_2>0$ you test $p_1-p_2>\Delta$.