I am working with a dataset including 65 participants, 35 in the control group and 30 in the experimental group. The objective is to assess wether the experimental group is non-inferior to the control group.

There are 2 events in the control group and 1 in the experimental group. Independent sample test of proportions shows that the experimental group is non-inferior to the control group.

However, because of the rare event rate in combination with the small sample size, I want to measure the stability of our results.

Does anybody know how I can measure the stability of the results of the non-inferiority test in this dataset.

Thank you in advance, Listeb

  • $\begingroup$ Would a higher underlying probability of an event make a group inferior or superior? What do you mean by stability? You might want to look at the possible patterns if each individual independently were to have a probability $\frac3{65}$ of an event, and might see that with that hypothesis what you actually observed was the second most likely outcome. $\endgroup$
    – Henry
    Dec 14, 2022 at 13:15
  • $\begingroup$ Thank you for your response. $\endgroup$
    – Listeb
    Dec 14, 2022 at 13:51
  • $\begingroup$ If an additional event occurs in the experimental group, as a consequence, with this sample size it cannot yet be proven that the experimental group is non-inferior. This does not make our results "stable".Is there a way for us to evaluate the stability of our findings? $\endgroup$
    – Listeb
    Dec 14, 2022 at 13:59

1 Answer 1


First, I ask you to think about the logic behind the notion that your data shows non-inferiority even though, if just one more (adverse) event had occurred among the experimental group, that finding would have been reversed. The difference between groups is almost the smallest possible difference. To say that it supports any strong conclusion about a substantial difference in the population is counter-intuitive, right? In one of your comments, if not in your original question, you seem to recognize this. And when I run a one-tailed test of proportions, I obtain p = .32.

Second, I encourage you to answer @Henry’s questions.

Third, you can use a test of proportions to create a confidence interval (CI) for the difference you obtained between group percentages. The CI’s soundness would depend on your method of sampling and on your choice of population to which you aim to generalize. Right now I’m seeing that the difference is -0.024 (-2.4 percentage points) which under the right assumptions would yield a 95% CI of [-0.126, 0.079]. Such a CI would say something about the range of expected results on resampling, which I think translates to “stability.”

Fourth, from another comment I understand that you want to know what-size difference, with what sample sizes, would produce findings with a given p-value, or with a narrow CI. You could learn this through trial and error with some software such as Excel; with a number of calculator websites; or working with a statistical power program such as the open-source G-Power.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.