2
$\begingroup$

I wanted to know what the difference between balanced and unbalanced design.

Firstly, here are two advantages of balanced designs:

The test will have larger statistical power, The test statistic is less susceptible to small departures from the assumption of equal variances (homoscedasticity).

Additionally, I found a source that says the following:

The main advantage (of balanced designs) is that the factors are independent and so the variance can be decomposed into the individual contributions without confounding.

My questions:

  1. Why do unbalanced designs have lower statistical power?
  2. Why are unbalanced designs susceptible to small departures from the assumption of equal variances?
  3. If it were a unbalanced design how would the variance be confounded?

Thanks.

$\endgroup$

1 Answer 1

2
$\begingroup$

You raise three quite different issues, perhaps better discussed separately. I will look at 'loss of power' first--and in the greatest detail.

Loss of power due to imbalance in a 2-sample t test. Suppose you have two populations $\mathsf{Norm}(\mu = 100, \sigma = 9)$ and $\mathsf{Norm}(\mu = 110, \sigma = 9),$ and you can afford 40 observations to see if you can detect the difference in population means using a pooled t test.

You have a choice between a balanced design with $n_1=n_2=20$ or an unbalanced design with $n_1 = 5$ and $n_2 = 35.$

Intuitively, the unbalanced design is worse because it spends a lot of resources discovering the nature of population 2, but not nearly enough getting information on population 1. "A chain is only as strong as its weakest link." The conclusion from the pooled t test cannot be better than the weak information on the first population.

In the simulation below, we see that the power of a t test from the balanced design is about 92% (so we're very likely to detect the difference in population means). By contrast, the power for the unbalanced design is only about 62%.

set.seed(515);  m = 10^5    
pv.u = replicate(m, 
       t.test(rnorm(5, 100, 9), rnorm(35, 110, 9), var.eq=T)$p.val )
mean(pv.u < .05)
[1] 0.61952

set.seed(515)
pv.b = replicate(m, 
       t.test(rnorm(20, 100, 9), rnorm(20, 110, 9), var.eq=T)$p.val )
mean(pv.b < .05)
[1] 0.92743

Unpredictable level of a pooled t test when variances are unequal. If population variances are unequal and the smaller sample corresponds to the larger variance, then the pooled test intended to be at level $\alpha$ may have a much different level than intended. However, this difficulty can be avoided by doing a Welch separate-variances) t test.

In the simulation below, population means are equal so the null hypothesis is true. However, what is intended to be pooled t test at the 5% level turns out to reject over 40% of the time. Some distortion of the significance level may occur even in less badly unbalanced designs. In any case, even for an unbalanced design, the Welch test has nearly the intended 5% significance level.

set.seed(515);  m = 10^5
pv.p = replicate(m, 
       t.test(rnorm(5, 100, 15), rnorm(35, 100, 3), var.eq=T)$p.val )
mean(pv.p < .05)
[1] 0.44354

set.seed(515)
pv.w = replicate(m, 
       t.test(rnorm(5, 100, 15), rnorm(35, 100, 3))$p.val )
mean(pv.w < .05)
[1] 0.05119

Confounding. Your third question has to do with ANOVA designs. I will only say that in many unbalanced designs the rows of the ANOVA table do not contain independent random variables, so that the so-called "F-statistics" don't really have F-distributions. Many kinds of errors--in addition to confounding--may result. Without access to your 'source' I can't know exactly what the author has in mind.

In general, it is usually best to use balanced designs where possible. Sometimes a design gets slightly out of balance because of operator error, faulty equipment, or subjects who exercise their rights to withdraw from a study. Then there are often reasonable ways to handle the imbalance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.