# What is the correct way to determine the amount of difference between two proportions?

Suppose I am testing the effectiveness of a new piece of machinery. The current machine has a success rate of 20%, that is, for every hundred widgets produced, 20 can be sold.

Now I've developed a new prototype and I want to know if it performs better. Additionally, it only makes sense to invest in the new machine if it is 10% better, i.e. a 22% success rate.

So I set up an experiment and collect this data:

machine     attempts   successes
---------------------------------
Original      n1         k1
Prototype     n2         k2


Assume that
* alpha= 5%
* beta= 20%
* I'd like to use the smallest sample size possible.

My understanding is that normally I'd want a 2-tailed difference of proportions test. But that test would only tell me if the two machines performed differently, not if one was 10% or more better.

How do I determine what sample size to use, what is the appropriate test(s) to use, and how do I report the magnitude of the difference?

• How large are $n_1$ and $n_2$? Tests for comparing proportions tend to lose power if the sample sizes differ too much. Jul 20, 2012 at 6:03
• If the new machine has a $21.8\%$ success rate, you would like to reject it unless the original really only succeeds about $19.8\%$ of the time? Does testing the original cost the same as testing the prototype? Jul 20, 2012 at 6:16
• @MånsT n1 and n2 could be between 1,000 to 100,000 if necessary. I can keep them roughly equal if that helps power.
– dan
Jul 20, 2012 at 13:46
• @Douglas Zare testing both machines costs the same. The business case is that the cost of fully developing the prototype would only pay off if I can expect a 10% improvement.
– dan
Jul 20, 2012 at 14:05
• @dan: I have some recommendations, but I don't have the time to post them now. I'll write an answer on Sunday or Monday unless threre already is a thorough answer by then! Jul 20, 2012 at 20:17

The standard hypothesis test only tells you that you are significantly better but not by the magnitude. To show that the magnitude is greater than a specified $\Delta$ requires a larger sample size. Instead of testing $p_1-p_2>0$ you test $p_1-p_2>\Delta$.