What is necessary sample size for B in A/B test with unequal sample sizes?

Question

I have a situation in which I want to measure if the error rate in system B (test) is worse than the error rate in system A (control). System A has a sample size of 100 and that cannot be modified. So my question is, how can I get an estimate of the minimum sample size required of system B to know if the error rate is worse than in system A?

To put some numbers in, let's say the mean error rate of system A is 1 and the standard deviation is 0.1. Let's assume I want 95% confidence and a power of 0.8. I'm assuming a one-sided test because I'm only interested in testing whether system B is worse than system A, not if it's better. I also assume a margin of error of sd/10=0.01.

Using R's power.t.test, I input the following:

power.t.test(sig.level = .05, delta = .01, sd = .1, alternative = 'one.sided', power = 0.8)

     Two-sample t test power calculation 

              n = 1237.188
          delta = 0.01
             sd = 0.1
      sig.level = 0.05
          power = 0.8
    alternative = one.sided

NOTE: n is number in *each* group

First, is power.t.test the right method to use here? Second, the result says I need 1237 samples in each group. However, since system A (control) cannot have more than 100 samples, how can I interpret or modify this test to account for that?

Answer 1

For an unequal sample size problem, you can use pwr.t2n.test from library(pwr).

I'm assuming from your question that you want to be able to detect a mean difference only .1 standard deviations away, so set d=.1. Unfortunately, I don't think you'll be able to achieve .8 power under the conditions you've described, as this produces an error message:

library(pwr)
pwr.t2n.test(n1=100, sig.level = .05,  d=.1, power=.8, alternative = 'great')

If you set a more modest goal of being able to detect a d=.5 standard deviation difference between A and B, you'd need a sample size of about 33 in system B.

pwr.t2n.test(n1=100, sig.level = .05,  d=.5, power=.8, alternative = 'great')

         n1 = 100
         n2 = 33.31148
          d = 0.5
  sig.level = 0.05
      power = 0.8
alternative = greater