# What is necessary sample size for B in A/B test with unequal sample sizes?

I have a situation in which I want to measure if the error rate in system B (test) is worse than the error rate in system A (control). System A has a sample size of 100 and that cannot be modified. So my question is, how can I get an estimate of the minimum sample size required of system B to know if the error rate is worse than in system A?

To put some numbers in, let's say the mean error rate of system A is 1 and the standard deviation is 0.1. Let's assume I want 95% confidence and a power of 0.8. I'm assuming a one-sided test because I'm only interested in testing whether system B is worse than system A, not if it's better. I also assume a margin of error of sd/10=0.01.

Using R's power.t.test, I input the following:

power.t.test(sig.level = .05, delta = .01, sd = .1, alternative = 'one.sided', power = 0.8)

Two-sample t test power calculation

n = 1237.188
delta = 0.01
sd = 0.1
sig.level = 0.05
power = 0.8
alternative = one.sided

NOTE: n is number in *each* group


First, is power.t.test the right method to use here? Second, the result says I need 1237 samples in each group. However, since system A (control) cannot have more than 100 samples, how can I interpret or modify this test to account for that?

For an unequal sample size problem, you can use pwr.t2n.test from library(pwr).

I'm assuming from your question that you want to be able to detect a mean difference only .1 standard deviations away, so set d=.1. Unfortunately, I don't think you'll be able to achieve .8 power under the conditions you've described, as this produces an error message:

library(pwr)
pwr.t2n.test(n1=100, sig.level = .05,  d=.1, power=.8, alternative = 'great')


If you set a more modest goal of being able to detect a d=.5 standard deviation difference between A and B, you'd need a sample size of about 33 in system B.

pwr.t2n.test(n1=100, sig.level = .05,  d=.5, power=.8, alternative = 'great')

         n1 = 100
n2 = 33.31148
d = 0.5
sig.level = 0.05
power = 0.8
alternative = greater