# What statistical test should I use to check the difference in a binary variable?

I want to test two different settings of some process which produces an output value based on a parametrized probability distribution (the exact distributions are unknown to me, but they are influenced by the setting). The final observable is whether the output value exceeds some threshold. Then I want to show that setting #1 is more likely to produce output values greater than the threshold than setting #2.

For example consider the following two distributions:

I will collect many samples for both settings independently and these will be either 1 or 0 based on whether they fall in the shaded region where $$x > threshold (= 3)$$. So I will obtain for example:

\begin{align} s_1 &= (1, 0, 1, 0, 0, 0, 1, 0, \ldots) \hspace{1cm} \textrm{Setting 1} \\ s_2 &= (0, 0, 0, 1, 1, 0, 0, 0, \ldots) \hspace{1cm} \textrm{Setting 2} \end{align}

Now I want to test whether setting #1 produced significantly more $$1's$$ than setting #2. I'm unsure which statistical test to use in this situation. I'd also like to understand how to estimate the minimum number of samples required to reach a predefined statistical significance level (e.g. if I can simulate the process with an approximation of the two distributions, would this help in estimating the minimum sample size)?

Edit: You can perform power analysis using this R package, in particular the function pwr.2p2n.test. Notice that the input to these functions includes only the probabilities of your values exceeding your threshold, so all you need to calculate from your sophisticated model is the expected frequency of 1's in each group under the minimal effect size you want to detect.
• Thanks, the binomial proportion test looks promising for what I want to do. Regarding the estimation of sample size, I do know how to simulate that process. I have a sophisticated model available which represents the actual device (process) accurately. My question was how I can use simulation results to estimate the min. sample size. Something like "When drawing N samples from both distributions, with 99.9% probability I want to reach a significance level of $10^{-3}$." How large should N be to achieve this? I could repeat doing this by varying $N$ but that seems very inefficient. Oct 6, 2020 at 12:24