How to estimate the minimum required sample size to reach a certain significance level for a binomial test? I'm about to conduct an experiment where I will measure a binary variable. I will test two different settings of a machine and then I want check whether these two settings yield different proportions in the observed binary variable by using a binomial test.
Obtaining samples from that machine is expensive (both in terms of time and money) so I would like to take as few samples as possible to reach the following goals:

*

*If the two settings actually yield different proportions (this is what I expect) then I want to reach a significance level of $p = 10^{-3}$ with the binomial test.

*I accept a maximum 1 out of 1,000 chance (i.e. 0.1%) that this significance level will not be reached due to unlucky sampling, even though the two proportions are actually different.

That means with a probability of 99.9% I want to reach a significance level of $10^{-3}$ when drawing $N$ samples. The question is how large should $N$ be to fulfill these requirements?
I have a numerical model of that machine which gives me access to the theoretical proportions under both settings (e.g. $p_1$ and $p_2$). I suppose these need to be used in order to estimate the minimum sample size $N$?
 A: This is a standard power analysis. To determine how many samples you need in each setting, you need to specify: a baseline value of interest, the difference between values that you want to detect, the chance that you are willing to accept a false-positive difference (Type-I error), and the chance that you are willing to accept that you miss a true-positive result (Type-II error). The power is 1 minus that Type-II error.
There are on-line tools for doing these calculations, for example here for your binomial two-sample test. Even with the stringent power (0.999) and Type-I error (0.001) requirements,* it's not hard to find a difference between $p_1=0.5$ and $p_2=0.99$ with a 2-sided test (which is what you should specify), 55 samples for each setting.
If instead you had wanted to distinguish $p_1=0.50$ from $p_2=0.55$ with the same specifications, you would have had a much harder time: 8113 samples each. It's easier to distinguish the same percentage-point difference near the ends of the binomial probability scale; to distinguish $p_1=0.94$ from $p_2=0.99$ you only need 1091 samples each. So use a proper power calculation instead of relying on gut instincts, as where you are along the probability scale and the difference you want to find matter a lot.

*Frequent choices are Type-I error of 0.05 and power of 0.8.
