# Experiment design: Can unbalanced dataset be better than balanced?

I'm on the stage of experiment design of some biomedical time-course study. Let's say we will have 2 groups of subjects - case and control. The total number of subjects is limited (for example, 30), and we need to decide how it's better to split subjects between case and control. One way is to have balanced dataset 50% of case vs. 50% of control. Another way is to assign 2/3 of subjects to case and 1/3 to control, since the case subjects would be more interesting.

What experiemnt design would you suggest? Which issues/pitfalls should be considered in both case? Should the experiment design depend on questions asked?

-
Are you assigning patients to cases or control, or merely collecting those who happen to be cases and controls? –  Fomite Nov 28 '12 at 2:42
@EpiGrad: Yes, we're assigning. But did I say patients? The question is more general. Subjects can be mice, for examples. Are you thinking about any ethical issues with humans? –  yuk Nov 28 '12 at 20:55
No, but I tend to read "biomedical" and head straight toward human subject. That's my own bias :) –  Fomite Nov 28 '12 at 20:58

How this is answered depends on how the data will be analyzed. But there are general principles that can be applied. The main idea is to identify what is under experimental control and vary that to optimize the ability of the statistical test to detect an effect if one indeed exists.

To illustrate how this sort of thing is worked out, let's consider the classic textbook situation evoked by the description in the question: a t-test to compare the treatment mean $\bar{X}_1$ to the control mean $\bar{X}_2$, assuming (at least approximately) equal variances $\sigma^2$ in the two groups. In this situation we can control the numbers of subjects placed into the control group ($n_1$ of them) and the treatment group ($n_2$ of them). The strategy is to do this in a way that maximizes the chances of detecting a difference between the group means in the population, if such a difference really exists. Along the way we are usually willing to make reasonable simplifying assumptions, expecting that the optimal design attained in such idealized circumstances will be close to optimal under true experimental conditions. Watch for those assumptions in the following analysis.

In the notation of the linked Wikipedia article, the $t$ statistic is

$$t = \frac{1}{\sqrt{1/n_1 + 1/n_2}}\frac{\bar{X}_1 - \bar{X}_2}{s_{12}}$$

where $s_1^2$ and $s_2^2$ are the usual variance estimates in the two groups and $$s_{12}^2 = \frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n-2}$$ estimates the common variance $\sigma^2$.

For assumed normal distributions within the groups and/or for sufficiently large $n_1$ and $n_2$, it is at least approximately the case that:

• $\bar{X}_1 - \bar{X}_2$, $(n_1-1)s_1^2/\sigma^2$, and $(n_2-1)s_2^2/\sigma^2$ are independent;

• the former has a Normal distribution with zero mean; and

• the latter have $\chi^2$ distributions with $n_1-1$ and $n_2-1$ degrees of freedom, respectively.

Whence the sum $(n_1-1)s_1^2 + (n_2-1)s_2^2$ will be $\sigma^2$ times a $\chi^2(n-2)$ distribution no matter what values $n_1$ and $n_2$ may take.

Accordingly, when the null hypothesis is true, $t$ will have a Student $t$ distribution with $n-2$ degrees of freedom regardless of how we choose $n_1$ and $n_2$. Our choice of $n_1$ and $n_2$ will not affect that. However, when the null hypothesis is false, the ratio $$\frac{\bar{X}_1 - \bar{X}_2}{s_{12}}$$ will tend to be near the standardized effect size--the size of the difference between the groups relative to $\sigma^2$. In computing $t$, that difference is magnified by the coefficient $$\frac{1}{\sqrt{1/n_1 + 1/n_2}}.$$ We obtain the greatest power to detect a difference when the magnification is as great as possible.

Given that $n_1+n_2=n,$ the "experimental factor" term

$$\frac{1}{\sqrt{1/n_1 + 1/n_2}} = \sqrt{\frac{1}{n}\left(n^2/4 - (n_1 - n/2)^2\right)}$$

obviously is maximized by making $(n_1-n/2)^2$ as small as possible (giving a maximum value close to $\sqrt{n}/2$.) Consequently, the best power to detect a difference between the treatment and control means with this test is attained when the subjects are divided as evenly as possible (but randomly) between the groups.

As a further demonstration of this result, I simulated the case where $n=16$, with Normal distributions within each group, using four standardized effect sizes $\delta = 0, 1/2, 1,$ and $3/2$. By running $N=5000$ replications of each experiment for all reasonable values of $n_1$ (ranging from $2$ through $n/2=8$), a Monte-Carlo distribution of $t$ was obtained. The following plots show how its mean value corresponds closely to the experimental factor except when there is no effect ($\delta=0$), exactly as claimed.

Because the real world is rarely as nicely behaved as a computer simulation, let's give our analysis a hard time by simulating highly skewed distributions within the groups (which is hard for a $t$ test to deal with). To this end I used a Gamma$(2, 1)$ distribution for the control group and a Gamma$(2, 1+\delta/2)$ distribution for the treatment group. Here are the results:

Although these plots are not as perfect as the preceding ones, they come close enough despite the substantial departures from the simplifying assumptions ($n$ is small and the data are nowhere near Normally distributed). Thus we can trust the conclusion that the experiment will be most powerful when subjects are close to equally divided between treatment and control groups.

For those who would like to study this further, here is the R code.

#
# Perform a t-test to compare the first n1 values to the next n2 values of x.
#
f <- function(x, n1, n2) t.test(x[1:n1], x[1:n2+n1], var.equal=TRUE)\$statistic
#
# Perform a full simulation for n1 = 2, 3, ..., n/2 for a given standardized
# effect size delta.  N Monte-Carlo iterations will be performed.
#
sim <- function(delta, n, N=500) {
z <- sapply(2:floor(n/2), function(n1) {
apply(matrix(rnorm(N*n, mean=c(rep(delta, n1), rep(0, n-n1))),  nrow=N, byrow=TRUE),
1, f, n1=n1, n2=n-n1)
})
apply(z, 2, mean) # Average t-statistics, one for each of the N iterations.
}
#
# Run simulations for various deltas and compare to the "experimental factor,"
# here stored in x.
#
n <- 16 # Total number of subjects in the experiment
x <- sapply(2:floor(n/2), function(x) 1/sqrt(1/x + 1/(n-x)))
par(mfrow=c(1, 4))
for (delta in c(0, 1/2, 1, 3/2)) {
y <- sim(delta, n) # N=5000 was used for the figure in the text
plot(x, y, pch=19, col="Red", main=paste("Delta =", delta)),
xlab="Experimental factor", ylab="Mean simulated t statistic")
abline(lm(y ~ x))  # Least squares fit for comparison
}


For the second simulation, the call to rnorm within sim was replaced by

rgamma(N*n, shape=2, scale=c(rep(1+delta/2, n1), rep(1, n-n1)))

-
Wow, thank you for so detailed answer. Very interesting. Certainly worth to be accepted. –  yuk Jan 25 at 4:24