I am designing a project which requires me to select a Control Group. One of the business requirements stipulates that I select as small a sample size as possible for the Control Group so there will be more records for the Test Group.
I would like to select a Control Group that is as small as possible and still be confident that the result would be statistically significant. Does anyone know which test would be appropriate for this?
As mentioned in comments, reducing the number of observations in one group will increase the required sample size in the other group, possibly leading to a prohibitive overall cost.
The problem of a prohibitive cost is not necessarily bound to happen though, if you can determine the specific costs associated with collecting observations in the control group and with collecting observations in the test group. However, if you're not in a position to determine costs for each group, this answer does not apply.
Once you determined the costs, you can use common sample size determination tools and methods (be it for power analysis or precision-based sample size determination), and search which allocation scheme other than 1:1 would minimize the overall cost of your study.
If you're using a tool that allows some scripting, you should be able to automate the search for an optimal allocation. At the end of this answer, I give an example in the programming language R, using a brute force approach (other approaches might be possible, but this one may be easier to understand). Here is the scenario and detailed approach, that should be also useful to those unfamiliar with R but familiar with programming:
Once we defined all these variables, we iterate over all possible number of observations in the control group, from 1 to 10,000. The reason for doing that is to determine, at each iteration, how many number of observations in the test group would be necessary to satisfy all the other conditions previously defined (power, effect size, alpha level, etc.), given the current number of observations in the control group.
Then, we can compute the overall cost of the study in this simulated scenario, given the number of observations we determined in each group. We store this overall cost in a list.
Finally, once the iterations are over, we find the smallest cost in the list, and the allocation scenario to which it corresponds. This is entirely possible to have different allocation scenarios leading to the smallest possible cost.
Given our conditions defined above, one solution to minimize the costs would be 1351 observations in the test group and 766 observations in the control group, leading to a total cost of \$36,490. However, another possible scenario could be 1318 vs. 777 observations. The total number of observations won't be the same (2117 vs. 2095), despite the total cost being identical in the two cases. Therefore, check if these different allocations could have other implications for you than statistical power or financial costs. A table or a plot of the required number of observations vs. the total cost may be informative:
| n1 | n2 | total cost |
|---|---|---|
| ... | ... | ... |
| 737 | 1452 | 36630 |
| 738 | 1448 | 36620 |
| ... | ... | ... |
| 765 | 1355 | 36500 |
| 766 | 1351 | 36490 |
| 767 | 1348 | 36490 |
| ... | ... | ... |
| 787 | 1291 | 36520 |
| ... | ... | ... |
Below is an example of code in R to find an allocation of sample sizes minimizing the costs. This code is just for illustration purposes, and does not retrieve all possible scenarios leading to the smallest cost, but just one of them.
library(pwr)
#Here we define the conditions
cost.test = 10
cost.control = 30
power = 0.95
alpha= 0.005
base.rate = 0.5
diff = 0.1
# We define a function returning the overall cost of the study, given a certain number of observations in the control group (n1)
compute.cost = function(n1){
## The pwr.2p2n.test function will return the required sample size for the control group.
result = tryCatch( pwr.2p2n.test(
h=ES.h(p1=base.rate, p2=base.rate+diff),
n1 = n1,
n2 = NULL,
sig.level = alpha,
power = power,
alternative = "two.sided"),
error = function(e) NULL)
if (is.null(result)){
#If there's no solution in the current scenario, the pwr.2p2n.test function returns an error. We capture the error if this is the case, and return an infinite cost.
return(Inf)
}
#If there is a solution in the current scenario, we compute the total cost of the study, and return it:
total.cost = cost.control * result$n1 + cost.test * ceiling(result$n2 )
return(total.cost)
}
possible.n1 = 1:10000
#We define a realistic set of values we could use as the sample size for the control group. Here, we'll test values from 1 to 10,000
costs = c() #we create a vector to store the calculated costs for each scenario
#We iterate over the possible values of n1 (1 to 10000), and compute the cost for each scenario
for (n1 in possible.n1){
cost = compute.cost(n1)
costs = c(costs, cost)
}
#Finally, we find the smallest possible cost, and retrieve the corresponding scenario
index.min.cost = which.min(costs)
n1 = possible.n1[index.min.cost]
n2= pwr.2p2n.test(h=ES.h(p1=base.rate, p2=base.rate+diff),
n1 = n1,
n2 = NULL ,
sig.level=alpha,
power = power,
alternative = "two.sided")$n2
print(paste0("You need ", n1, " observations in the first group and ", ceiling(n2), " observations in the second group to minimize the costs of the study. The overall cost will be $", min(costs)))
* It's possible to have more complicated situations, for instance where the cost of one observation is not fixed but changes as a function of the total number of observations. Things are kept simple here, but the overall approach shouldn't change in those more complicated scenarios.
