I am trying to understand why testing for non-inferiority requires pretty much the same sample size as testing for superiority (I assume the latter is the same as a one-sided test for a given MDE).

I was asked how big a sample we need to test that a certain change to our website's backend has no (negative) effect on visitor conversion. I said that it should be easier than testing for a lift in conversion. But that doesn't seem to be the case.

Running a one-sided test with 95% confidence and 90% power and assuming a 9% conversion rate and a 5% effect (0.45% lift) requires some 70k examples: http://powerandsamplesize.com/Calculators/Compare-2-Proportions/2-Sample-1-Sided

At the same time, running a non-inferiority test with a 0.45% margin requires 69k samples http://powerandsamplesize.com/Calculators/Compare-2-Proportions/2-Sample-Non-Inferiority-or-Superiority

Is that right or am I missing something?


1 Answer 1


What alternative scenario are you assuming? If you assume the same positive true effect, then superiority will require a smaller sample size for the same power. If you assume no effect, then superiority testing does not even make sense.

On the other hand, if you look at superiority assuming a true improvement by x or non-inferiority with a non-inferiority margin of x assuming there is in truth no difference, then you get about the same probability of a significant result under the alternative hypothesis with the same sample size.

  • $\begingroup$ I guess I am still confused why the following post says you need a lot less data for a non-inferiority test: blog.analytics-toolkit.com/2017/… There is a table in the middle. $\endgroup$
    – iggy
    Commented Apr 2, 2018 at 4:26
  • $\begingroup$ The table in the blog assumes that there is in truth an improvement of 5 percentage points and under that assumption non-inferiority with a margin of 2 percentage points is easier to show than superiorty. Very approximately non-inferiority is as easy to show as superiorty assuming that the true difference is 5+2=7 percentage points (and sample size is about proportional to effect size squared i.e. $5^2/7^2\approx 0.5$). $\endgroup$
    – Björn
    Commented Apr 2, 2018 at 5:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.