0
$\begingroup$

I want to test two different settings of some process which produces an output value based on a parametrized probability distribution (the exact distributions are unknown to me, but they are influenced by the setting). The final observable is whether the output value exceeds some threshold. Then I want to show that setting #1 is more likely to produce output values greater than the threshold than setting #2.

For example consider the following two distributions:

Example

I will collect many samples for both settings independently and these will be either 1 or 0 based on whether they fall in the shaded region where $x > threshold (= 3)$. So I will obtain for example:

$$ \begin{align} s_1 &= (1, 0, 1, 0, 0, 0, 1, 0, \ldots) \hspace{1cm} \textrm{Setting 1} \\ s_2 &= (0, 0, 0, 1, 1, 0, 0, 0, \ldots) \hspace{1cm} \textrm{Setting 2} \end{align} $$

Now I want to test whether setting #1 produced significantly more $1's$ than setting #2. I'm unsure which statistical test to use in this situation. I'd also like to understand how to estimate the minimum number of samples required to reach a predefined statistical significance level (e.g. if I can simulate the process with an approximation of the two distributions, would this help in estimating the minimum sample size)?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

The distribution of the number of 1's in each group is a binomial distribution, since it's a count of iid failures/successes. You can find information about the adequate statistical rest here. You can easily simulate this process: just think about the number of samples from each group and the probabilities of getting a 1 from each group and use these parameters to simulate a binomial distribution.

Edit: You can perform power analysis using this R package, in particular the function pwr.2p2n.test. Notice that the input to these functions includes only the probabilities of your values exceeding your threshold, so all you need to calculate from your sophisticated model is the expected frequency of 1's in each group under the minimal effect size you want to detect.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks, the binomial proportion test looks promising for what I want to do. Regarding the estimation of sample size, I do know how to simulate that process. I have a sophisticated model available which represents the actual device (process) accurately. My question was how I can use simulation results to estimate the min. sample size. Something like "When drawing N samples from both distributions, with 99.9% probability I want to reach a significance level of $10^{-3}$." How large should N be to achieve this? I could repeat doing this by varying $N$ but that seems very inefficient. $\endgroup$
    – a_guest
    Oct 6, 2020 at 12:24
  • $\begingroup$ @a_guest I edited my answer in response to this comment. The R package I linked addresses exactly your concern, and its inputs are only the probabilities of finding a 1 in each group. $\endgroup$
    – PedroSebe
    Oct 6, 2020 at 14:20
  • $\begingroup$ Thanks, that package looks very helpful indeed. Do you know how it computes the sample size? Does it rely on some formulas or does it infer the sample size from a parameter scan for example? $\endgroup$
    – a_guest
    Oct 7, 2020 at 15:40
  • 1
    $\begingroup$ @a_guest It probably uses formulas derived from the normal approximation to the gaussian. The statistical power of the test is a function of the population parameters, so you have to provide some estimate for them (it can be an educated guess, since you don't have the real data yet). Keep in mind that the difference between the binomial parameters in the groups should be compatible with the smallest difference you want to be able to measure. The smaller the difference, the larger the necessary sample size. You can find more info about this searching for statistical power analysis $\endgroup$
    – PedroSebe
    Oct 7, 2020 at 17:18
0
$\begingroup$

Logistic regression is suitable to the analysis of binary data, wherein you can have binary independent as well as dependent variables. SPSS, and R are good options to do this. Chi square might also be an option here.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.