# What is the correct way to determine the amount of difference between two proportions?

Suppose I am testing the effectiveness of a new piece of machinery. The current machine has a success rate of 20%, that is, for every hundred widgets produced, 20 can be sold.

Now I've developed a new prototype and I want to know if it performs better. Additionally, it only makes sense to invest in the new machine if it is 10% better, i.e. a 22% success rate.

So I set up an experiment and collect this data:

machine     attempts   successes
---------------------------------
Original      n1         k1
Prototype     n2         k2


Assume that
* alpha= 5%
* beta= 20%
* I'd like to use the smallest sample size possible.

My understanding is that normally I'd want a 2-tailed difference of proportions test. But that test would only tell me if the two machines performed differently, not if one was 10% or more better.

How do I determine what sample size to use, what is the appropriate test(s) to use, and how do I report the magnitude of the difference?

-
How large are $n_1$ and $n_2$? Tests for comparing proportions tend to lose power if the sample sizes differ too much. –  MånsT Jul 20 '12 at 6:03
If the new machine has a $21.8\%$ success rate, you would like to reject it unless the original really only succeeds about $19.8\%$ of the time? Does testing the original cost the same as testing the prototype? –  Douglas Zare Jul 20 '12 at 6:16
@MånsT n1 and n2 could be between 1,000 to 100,000 if necessary. I can keep them roughly equal if that helps power. –  dan Jul 20 '12 at 13:46
@Douglas Zare testing both machines costs the same. The business case is that the cost of fully developing the prototype would only pay off if I can expect a 10% improvement. –  dan Jul 20 '12 at 14:05
@dan: I have some recommendations, but I don't have the time to post them now. I'll write an answer on Sunday or Monday unless threre already is a thorough answer by then! –  MånsT Jul 20 '12 at 20:17
show 1 more comment

## 1 Answer

You can do a one-sided test but even if you do a two-sided test the side to which you exceed the threshold clearly tells you if you are better or worse.

The standard hypothesis test only tells you that you are significantly better but not by the magnitude. To show that the magnitude is greater than a specified $\Delta$ requires a larger sample size. Instead of testing $p_1-p_2>0$ you test $p_1-p_2>\Delta$.

-
add comment