4
$\begingroup$

I am new to power analysis and have been experimenting with one-way fixed effects Anova in G*Power (Fixed effects, omnibus, one-way, Post hoc power).

When dealing with unbalanced design, the effect size is affected. When power of such a test is computed, is it also affected by unequal sample sizes (I mean besides the effect size)?

$\endgroup$
4
  • $\begingroup$ Could you clarify if your question is about how to conduct a power analysis (i.e., how many observations should I collect?), or is your question about how to conduct ANOVA (i.e., does it matter if my group sizes are different?). In other words, have you already collected your data, or are you making a plan to collect data? $\endgroup$
    – 5ayat
    May 17, 2016 at 15:07
  • 1
    $\begingroup$ I would like to know how power calculation for an unbalanced design differs from calculation for a balanced one (fixed one-way Anova, 4 groups). For those familiar with G*Power I can further explain where my question stems from: the program allows to compute an effect size given unequal sample sizes; when the computed effect size is used to calculate power, and given that total sample size is not a multiple of the number of groups, the program bases its calculation on average group size. I'd like to know whether unequal sample sizes directly affect the power calculation. $\endgroup$
    – Gregory
    May 17, 2016 at 18:07
  • 1
    $\begingroup$ An unbalanced design would not affect most of what we call "effect sizes". The whole point of designating an effect size was to prize apart meaningful inferences from overpowered analyses - i.e. my drug is only useful if it lowers blood pressure by 10 mmHg compared to control, regardless of $p$. $\endgroup$
    – AdamO
    Jan 24 at 12:45
  • 1
    $\begingroup$ There are some cases when an imbalanced design can be more powerful than a balanced one, even in case of simple random sampling with 1:1 randomization. $\endgroup$
    – AdamO
    Jan 24 at 12:47

3 Answers 3

4
$\begingroup$

There is an easy rule of thumb for ANOVA: Unequality in sample sizes deteriorates the power. The reason is that the power depends mostly on the variance of the effect estimator, i.e. the mean differences between the groups. These mean differences have the least variance (given a total sample size) if all the sample sizes are equal.

$\endgroup$
3
  • $\begingroup$ Thank you for your answer @Horst Grünbusch. GPower outputs a message whenever the total sample size is not a multiple of the number of groups, saying that "the analysis was based on the average group size". Does this imply that the GPower calculation is correct only for samples of equal size? $\endgroup$
    – Gregory
    Jun 2, 2016 at 15:00
  • $\begingroup$ @Gregory that sounds correct to me. Such design assumptions are typical in canned software like G*power and R's pwr package as it simplifies the equations and user inputs $\endgroup$ Jan 23 at 19:02
  • $\begingroup$ @philchalmers You can take different group sizes into account in G*Power or in R packages like pwr. You do that by specifying the correct effect size $f$, whose formula and resulting value are different for the case of groups of different sizes. If you're interested, I provide more details in my answer below. $\endgroup$
    – J-J-J
    Jan 24 at 12:10
3
$\begingroup$

The quick and dirty way to solve for unequal sample size problems would be to just run a simulation. Below is one such example comparing equal groups and two other potential setups using R's SimDesign package: a control group with 40% of the total sample size, and a control group with 70% of the total size for a $k=4$ setting.

library(SimDesign)

# equal and unequal group designs
Design <- createDesign(N = 30 * 4,
                       f2=.3^2,
                       props = list(rep(1/4, 4), 
                                 c(.4, .2, .2, .2),
                                 c(.7, .1, .1, .1)))

#-------------------------------------------------------------------

Generate <- function(condition, fixed_objects = NULL) {
    Attach(condition)
    n.each <- N*props
    # stop if groups have fractions
    stopifnot(all.equal(n.each, as.integer(n.each))) 
    group <- factor(rep(1:4, times=n.each))
    eta2 <- 1 / (1/f2 + 1)
    
    # pick a nice SSG from mu=0 deviation form, and solve for SSE
    gmeans <- c(-2, -1, 1, 2)
    SSG <- sum(n.each * gmeans^2)
    SSE <- SSG*(1/eta2 - 1)
    sde <- sqrt(SSE / N)
    DV <- rep(gmeans, times=n.each) + rnorm(N, sd=sde)
    dat <- data.frame(DV=DV, group=group)
    dat
}

Analyse <- function(condition, dat, fixed_objects = NULL) {
    # Fit ANOVA + extract p-value
    aov <- aov(DV ~ group, dat)
    so <- summary(aov)
    ret <- c(p = so[[1]]$`Pr(>F)`[1])
    ret
}

Summarise <- function(condition, results, fixed_objects = NULL) {
    ret <- EDR(results) # empirical detection/rejection rate
    ret
}

#-------------------------------------------------------------------

res <- runSimulation(design=Design, replications=100000,
                        generate=Generate, analyse=Analyse, summarise=Summarise)
subset(res, select=N:p)

# A tibble: 3 × 4
      N    f2 props                      power     
  <dbl> <dbl> <lst>
1   120  0.09 [0.25, 0.25, 0.25, 0.25] 0.78252
2   120  0.09 [0.4, 0.2, 0.2, 0.2]     0.75352
3   120  0.09 [0.7, 0.1, 0.1, 0.1]     0.51941

Some of the more annoying features of using standardized effect sizes (like $f^2$) is that you'll often need to work out some of the translation equations to get something useful, such as converting from $f^2$ to $$\eta^2 = \frac{1}{1/f^2 + 1}$$

so that you can later work with the usual $SS_{total} = SS_{between} + SS_{within}$ and generate data from that structure by solving, for example, $$SS_{within} = SS_{between}(1/\eta^2 -1)$$ Otherwise, if you know the raw means + standard deviations for each group those could simply be substituted into the simulation, which I personally find more natural in empirical settings.

The true power assuming equal sample sizes can be found via the pwr package, which corresponds to the first row of the above output.

# equal groups
pwr::pwr.anova.test(k=4, n=30, f=.3)


     Balanced one-way analysis of variance power calculation 

              k = 4
              n = 30
              f = 0.3
      sig.level = 0.05
          power = 0.7808169

NOTE: n is number in each group

So for control groups that reflect the bulk of the sample size power can drop pretty substantially (hence why equal sample size conditions are ideal).

$\endgroup$
2
  • 1
    $\begingroup$ I'm not sure I understand fully your whole R code, but it looks like the effect size $f$ is fixed. It shouldn't be the case if you're comparing an "equal allocation" scenario to an "unequal allocation" scenario, as $f$ is partly a function of the number of observation in each group (i.e. an equal number of observations in each group can lead to a different effect size $f$ than an unequal number of observations in each group – keeping everything else identical). $\endgroup$
    – J-J-J
    Jan 24 at 12:41
  • 1
    $\begingroup$ @J-J-J that's correct, this simulation example holds $f$ constant and adjusts the within-subject variability term to accommodate to demonstrate the effect of having the same population $f$ across experiments but different sample sizes. The approach you presented makes more sense though as allowing $f$ to change as a function of the group proportions is more realistic. Also, good to know that G*Power does have support for unequal groups; thanks for sharing! $\endgroup$ Jan 24 at 13:44
3
$\begingroup$

The answer by Horst Grünbusch mentions a rule of thumb where power decreases with unequal sample sizes, but it is not necessarily true in all situations.

You might gain power if you affect a larger sample size to a group whose mean, compared to other groups, is discrepant from the mean of means. By doing so, you can increase the effect size - and if you increase the effect size, you increase the power to detect this effect.

Below, I give the formulas that lead to this conclusion, followed by a worked example and its application in G*Power, and finally a verification of the result by simulation in R.

For more details about the formulas and use cases where unequal sample sizes may be useful, you may be interested in Jacob Cohen's book on power analysis (Cohen, 1988, pp. 274-287; see references below).


The effect size $f$ is defined as:

$f=\frac{\sigma_m}{\sigma}$

where:

  • $\sigma$ is the standard deviation within each group (we assume equal variance),
  • $\sigma_m= \sqrt{\sum_{i=1}^k w_i(m_i-m)^2}$

For the calculation of $\sigma_m$:

  • $k$ is the number of groups,
  • $m_i$ is the group $i$ mean,
  • $w_i$ is the weight of the group $i$ such as $w_i = \frac{n_i}{N}$, $n_i$ being the number of observations in the group, and $N$ the total number of observations,
  • and $m$ is the mean of means, weighted per $w_i$, such as:

$m=\frac{\sum_{i=i}^kn_i m_i}{N}= \sum_{i=i}^k w_i m_i$

G*Power has a hidden function to calculate the correct effect size for unequal group sizes; it is available when you click the "determine" button next to the "effect size" field. We can use that to check if it is consistent with the formula above.

First, let's just look what happens with 3 groups whose means are 10, 7, and 7, with a standard deviation $\sigma$ of 5. If we allocate a total of 120 observations equally between the groups (i.e. 40 observation in each group), we have $m = 8$. Then:

$\sigma_m = \sqrt{\frac{1}{3} (10-8)^2 + \frac{1}{3} (7-8)^2 + \frac{1}{3} (7-8)^2} = 1.414 $

So we should have an effect size $f$ such as:

$f = \frac{\sigma_m}{\sigma} \approx 0.28$

With an alpha of 0.05, our power should be about 0.79.

Now, let's have a look at the case of an unequal allocation of observations between groups.

With weights of 0.5, 0.25, and 0.25 (that is, 60 observations in the first group, and 30 observations in each of the remaining groups), we have the following weighted mean:

$m = 10 \frac{60}{120} + 7 \frac{30}{120} + 7 \frac{30}{120} = 8.5$

Consequently:

$\sigma_m = \sqrt{0.5 (10-8.5)^2 + 0.25 (7-8.5)^2 + 0.25 (7-8.5)^2} = 1.5 $

and our effect size $f$ is:

$f = \frac{\sigma_m}{\sigma} = 0.3$

With such an effect size, with an alpha level of 0.05, our power would increase from 0.79 to about 0.84.

Let's see in G*power if those calculations are correct. After accessing the "determine effect size" menu, we first enter the data for the equal allocation scenario with the values given above (10, 7, and 7 for the group means, 5 for the standard deviation, and 40 observations in each group):

A screenshot of the 'determine effect size' menu for Gpower, with an equal allocation of observations between groups

The resulting effect size is about 0.28, similar to the result given by the previous calculations.

Now entering the data for the unequal group size scenario, we just change the allocation of observations to 60, 30, and 30, keep the other values untouched, and then click the "calculate" button:

Another screenshot of the 'determine effect size' menu for Gpower, with an unequal allocation of observations between groups

This time, we got an effect size of 0.3, like we found with the previous calculations.

Hence, in this specific scenario, it seems we would increase power if we used unequal group sizes.

But if we're not very confident in those formulas or in G*Power, we can double check that by simulation. In R, we can use the following code:

N= 120 #total sample size
props.equal = c(1/3, 1/3, 1/3) #proportions for the equal allocation scenario
props.unequal = c(0.5, 0.25, 0.25) #for the unequal allocation scenario
n.unequal = N*props.unequal 
n.equal = N*props.equal
sd = 5
# creating vectors where we'll store the pvalues generated by each simulation
pvalues.equal = c() 
pvalues.unequal = c()
# setting a seed for reproducibility
set.seed(0)
# we run 10000 simulations
for (i in 1:10000) {
  # simulating data for the equal allocation case
  group.equal = c(rep("group1", n.equal[1]), rep("group2", 
       n.equal[2]), rep("group3", n.equal[3]))
  df.equal = data.frame(group=group.equal)
  df.equal$value = ifelse(test = df.equal$group == "group1", 
                            rnorm(mean=10, sd=sd, n=N), 
                            rnorm(mean=7, sd=sd, n=N))
  # fitting anova on the data from this simulated "equal allocation" scenario
  res.equal = anova(aov(value ~ group, data = df.equal))
  # extracting and storing the p-value
  p.equal = res.equal$`Pr(>F)`[1]
  pvalues.equal = c(pvalues.equal, p.equal)
  
  # simulating data for the unequal allocation case
  group.unequal = c(rep("group1", n.unequal[1]), rep("group2", 
        n.unequal[2]), rep("group3", n.unequal[3]))
  df.unequal = data.frame(group=group.unequal)
  df.unequal$value = ifelse(test = df.unequal$group == "group1", 
                          rnorm(mean=10, sd=sd, n=N), 
                          rnorm(mean=7, sd=sd, n=N))
  # fitting anova on the data from the "unequal allocation" scenario
  res.unequal = anova(aov(value ~ group, data = df.unequal))
  # extracting and storing the p-value
  p.unequal = res.unequal$`Pr(>F)`[1]
  pvalues.unequal = c(pvalues.unequal, p.unequal)
}

# power = % of times the p-value from the anova was < 0.05 
power.equal = sum(pvalues.equal < 0.05)/length(pvalues.equal) 
power.unequal = sum(pvalues.unequal < 0.05)/length(pvalues.unequal)
print(paste0("Equal allocation: power of ", power.equal))
print(paste0("Unequal allocation: power of ", power.unequal))
> "Equal allocation: power of 0.7844"
> "Unequal allocation: power of 0.8402"

These results (0.7844 for equal allocation and 0.8402 for unequal allocation) are in line with the results from G*power and our previous calculations.

Of course, when planning a study and calculating the required sample size to reach a given power, making a mistake relative to the real differences between the groups could result in decreased power - in particular if you allocate more observations to a group those mean is in reality close to the weighted mean of means. So testing a variety of plausible scenarios is certainly a good idea when conducting a priori power analysis.

Note that using simulation gives you more flexibility, e.g. you can try to assign different standard deviations to each group to see how it affects power; you could also add a few additional lines of code to test a large range of plausible means for each group (I suspect that trying to do that with G*power would take you much more time).

References

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd edition). Routledge.

Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191. https://www.psychologie.hhu.de/fileadmin/redaktion/Fakultaeten/Mathematisch-Naturwissenschaftliche_Fakultaet/Psychologie/AAP/gpower/GPower3-BRM-Paper.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.