Non-inferiority margin and minimum detectable effect vs sample size

Question

I am trying to understand why testing for non-inferiority requires pretty much the same sample size as testing for superiority (I assume the latter is the same as a one-sided test for a given MDE).

I was asked how big a sample we need to test that a certain change to our website's backend has no (negative) effect on visitor conversion. I said that it should be easier than testing for a lift in conversion. But that doesn't seem to be the case.

Running a one-sided test with 95% confidence and 90% power and assuming a 9% conversion rate and a 5% effect (0.45% lift) requires some 70k examples: http://powerandsamplesize.com/Calculators/Compare-2-Proportions/2-Sample-1-Sided

At the same time, running a non-inferiority test with a 0.45% margin requires 69k samples http://powerandsamplesize.com/Calculators/Compare-2-Proportions/2-Sample-Non-Inferiority-or-Superiority

Is that right or am I missing something?

Answer 1

What alternative scenario are you assuming? If you assume the same positive true effect, then superiority will require a smaller sample size for the same power. If you assume no effect, then superiority testing does not even make sense.

On the other hand, if you look at superiority assuming a true improvement by x or non-inferiority with a non-inferiority margin of x assuming there is in truth no difference, then you get about the same probability of a significant result under the alternative hypothesis with the same sample size.