I have a situation in which I want to measure if the error rate in system B (test) is worse than the error rate in system A (control). System A has a sample size of 100 and that cannot be modified. So my question is, how can I get an estimate of the minimum sample size required of system B to know if the error rate is worse than in system A?
To put some numbers in, let's say the mean error rate of system A is 1 and the standard deviation is 0.1. Let's assume I want 95% confidence and a power of 0.8. I'm assuming a one-sided test because I'm only interested in testing whether system B is worse than system A, not if it's better. I also assume a margin of error of sd/10=0.01.
Using R's power.t.test, I input the following:
power.t.test(sig.level = .05, delta = .01, sd = .1, alternative = 'one.sided', power = 0.8)
Two-sample t test power calculation
n = 1237.188
delta = 0.01
sd = 0.1
sig.level = 0.05
power = 0.8
alternative = one.sided
NOTE: n is number in *each* group
First, is power.t.test
the right method to use here? Second, the result says I need 1237 samples in each group. However, since system A (control) cannot have more than 100 samples, how can I interpret or modify this test to account for that?