Suppose I am testing the effectiveness of a new piece of machinery. The current machine has a success rate of 20%, that is, for every hundred widgets produced, 20 can be sold.
Now I've developed a new prototype and I want to know if it performs better. Additionally, it only makes sense to invest in the new machine if it is 10% better, i.e. a 22% success rate.
So I set up an experiment and collect this data:
machine attempts successes --------------------------------- Original n1 k1 Prototype n2 k2
* I'd like to use the smallest sample size possible.
My understanding is that normally I'd want a 2-tailed difference of proportions test. But that test would only tell me if the two machines performed differently, not if one was 10% or more better.
How do I determine what sample size to use, what is the appropriate test(s) to use, and how do I report the magnitude of the difference?