# Test vs control group - how to increase the accuracy of my estimate?

I would like to find out if there is a way to reduce the statistical noise from my estimate of incremental value added from a certain customer treatment. Assume we have a group of clients for which we want to increase our revenue. We schedule some treatment (e.g. an ad or a special offer) for part of the customers ('test group') and don't provide any special treatment for the remaining customers ('control group'). The clients are assigned to test and control groups randomly.

One way to calculate the value added from such action is:

$$y=N_T*(\overline{x}_T-\overline{x}_C)$$,

where $$N_T$$ is the number of clients in the test group, $$\overline{x}_T$$ and $$\overline{x}_C$$ are the mean revenues per client in test and control group respectively and $$y$$ is the incremental value added from the treatment.

Assuming $$y$$ is significantly greater than 0 at some $$\alpha$$, is there a way to reduce the uncertainty around estimate $$y$$?

• Increase the sample size? Commented Jul 7, 2019 at 17:38
• Yes, that would help. However there is a trade off between the incremental value and the size of the control group. If we assign more customers to the control group, we limit the impact of our treatment, which in turn reduces the added value. Commented Jul 7, 2019 at 19:24
• Just to clarify, the sample size (customer group) is fixed, we cannot increase it. Commented Jul 7, 2019 at 19:36
• Then the power against a particular alternative would be largest if half of the available subjects were randomized to each group. This might not be the 'tradeoff' you want to make, but (roughly speaking) the smaller sample size has the greatest influence on power. Commented Jul 7, 2019 at 23:35
• Can you control for pre-intervention covariates? That should reduce noise Commented Jul 8, 2019 at 0:20

Comment continued: On the efficiency of balanced designs.

In the following example, we have a total of a thousand subjects available and hope to detect a difference of of 2 units between population means of two groups, each with standard deviation $$\sigma = 10.$$

Using a balanced two-sample design with sample sizes $$n_1 = n_2 = 500,$$ the power of a test at significance level 5% is about 89%.

By contrast, the power for an unbalanced design with $$n_1 = 100, n_2 = 900$$ is only about 47%. That's worse power than with a balanced design using only 200 subjects in each group (about 51%).

m = 10^5;  mu1 = 100;  mu2 = 102;  sg = 10

# balanced
n1 = 500;  n2 = 500
set.seed(707)  # for reprocibility
pv.bal = replicate(m, t.test(rnorm(n1,mu1,sg), rnorm(n2,mu2,sg), var.eq=T)$p.val) mean(pv.bal <= .05) [1] 0.886 # aprx power # unbalanced n1 = 100; n2 = 900 pv.ub = replicate(m, t.test(rnorm(n1,mu1,sg), rnorm(n2,mu2,sg), var.eq=T)$p.val)
mean(pv.ub <= .05)
[1] 0.4736     # aprx power

# Smaller balanced
n1 = 200;  n2 = 200
set.seed(707)
pv.sm = replicate(m, t.test(rnorm(n1,mu1,sg), rnorm(n2,mu2,sg), var.eq=T)\$p.val)
mean(pv.bal <= .05)
[1] 0.5127      # aprx power


Note: Various statistical softwares have procedures for finding power. Also, some online power calculators are reliable.

The theory is not difficult: Under the alternative that the discrepancy $$|\mu_1-\mu_2| = \delta,$$ the $$T$$-statistic has a non-central t distribution with $$\nu= n_1 + n_2 - 2$$ degrees of freedom, and non-centrality parameter $$\lambda = \frac{\delta}{\sigma\sqrt{1/n_1 + 1/n_2}}.$$ Thus the power of the two sided, pooled t test at the 5% level, in the unbalanced case $$n_1=100, n_2=900,$$ is computed in R as 47.4%.

nu = 998; t.c = qt(.975, nu)
del = 2; sg = 10; lam =  del/(sg*sqrt(1/100 + 1/900))
1 - diff(pt(c(-t.c, t.c), nu, lam))
[1] 0.4743743