1
$\begingroup$

I want to be able to calculate power for ANOVA with unequal sample sizes. I'm able to calculate one-way ANOVA (does not assume equal sample sizes) & effect size. Now just need to calculate power. I notice the pwr.anova.test() seems to assume equal sample size:

pwr.anova.test(k = NULL, n = NULL, f = NULL, sig.level = 0.05, power = NULL)

Arguments:
k Number of groups
n Number of observations (per group). f Effect size
sig.level Significance level (Type I error probability)
power Power of test (1 minus Type II error probability)

Is there an R code for Anova POWER for unequal sample sizes?

$\endgroup$
2
  • 2
    $\begingroup$ Asking for code is generally off topic here. Your post seems to imply that you have already run an ANOVA & now want to calculate power. That doesn't make sense. Once a test has been conducted, power plays no role. This sounds like bad old practices around 'post-hoc power'. FWIW, the easiest way to do this is to simulate. Search the site, the topic has been covered many times already. $\endgroup$ Dec 25, 2021 at 22:54
  • $\begingroup$ As noted above, you shouldn't be conducting post-hoc power analysis. However, if you're planning a study, you can absolutely use the pwr package for finding power for a range of pre-established scenarios (unequal vs. equal sample sizes). For the n parameter you give the average number of observations in each group, and very importanly, for the parameter f, you calculate correctly the effect size f, which is different if the group sample sizes will be balanced or unbalanced. I explain how to calculate correctly the effect size here: stats.stackexchange.com/a/637646/164936 . $\endgroup$
    – J-J-J
    Jan 30 at 7:39

1 Answer 1

1
$\begingroup$

The 'power and sample size' procedures for one-way ANOVAs usually give answers for models in which the number of replications at each level of the factor is equal. I suppose this is for simplicity and because the most efficient use of resources (to pay for replications) is usually a 'balanced' design with equal number of replications per level.

However, in some circumstances it is appropriate or necessary to have unequal numbers of replications for various levels. Examples: (a) You may be testing three new drugs against a current standard drug and want more replications for the standard drug for better power in ad hoc tests of each new drug against the standard. (b) Replications at some levels of the factor may be much more expensive than other levels.

When choosing numbers of replications for an ANOVA, it is worthwhile to anticipate that one may find that some differences are significant. Then it is a good idea to have sufficient power for useful ad hoc tests.

If you go sufficiently deeply into the formulas for power against different patterns of alternatives, you may be able to get exact results for power with differing numbers of replications. Perhaps more simply, you can get reasonably accurate power values by simulation based on unbalanced designs; I show one such simulation below.

Suppose you have an experiment with normal samples with population means $50, 54, 57,$ standard deviations $4, 6, 7$ and sample sizes $20, 15, 15.$ Then we show the R procedure oneway.test, which does not assume equal variances for the levels. You might have a dataset similar to the fictitious one below.

set.seed(1225)
x1 = rnorm(20, 50, 4)
x2 = rnorm(15, 54, 6)
x3 = rnorm(15, 57, 7)
x = c(x1, x2, x3)
g = rep(1:3, c(20,15,15))
oneway.test(x ~ g)

        One-way analysis of means 
        (not assuming equal variances)

data:  x and g
F = 11.774, num df = 2.000, denom df = 27.597, p-value = 0.0002009

So, this particular dataset to the specifications above shows significance at below the 1%. A simulation of 100,000 datasets will give a good idea of the power of such a design. [Notice that denom df < 47 above, because the three sample variances differ.]

set.seed(2021)
m = 10^5;  pv = numeric(m)
g = rep(1:3, c(20,15,15))
for (i in 1:m) {
 x1 = rnorm(20, 50, 4)
 x2 = rnorm(15, 54, 6)
 x3 = rnorm(15, 57, 7)
 x = c(x1, x2, x3)
 pv[i] = oneway.test(x~g)$p.val
 }
mean(pv <= 0.05)
[1] 0.90488             # aprx power
2*sd(pv <= 0.05)/sqrt(m)
[1] 0.001855511         # aprx 95% margin of sim error

The power of oneway.test at the 5% level of significance for this design is about $0.905 \pm 0.002.$

Notes: (1) I do not know of an exact formula for the power of oneway.test, which has different denominator degrees of freedom for each dataset, based on the observed variances of the various levels. See R documentation.

(2) A for-loop is not the most elegant way to program the simulation in R, but it seems to be among the easiest for beginners to understand.

$\endgroup$
3
  • 1
    $\begingroup$ The .90 power, given the not huge sample sizes suggests that one.way is a high powered test. That this is so is probably due to combination of 1. the broad alternative being any difference in means, and 2. your simulation using rnorm which of course satisfies the normality assumptions of one.way. Is this fair commentary on the .90 simulated power? $\endgroup$ Dec 26, 2021 at 9:59
  • 1
    $\begingroup$ I chose sample sizes I thought would be just large enough to give good power in view of relatively small population SDs and differences among 3 level population means about $\sigma/2.$ (I've dealt with similar situations before.) // Almost all 'power and sample size' computations for ANOVAs assume normality. If populations are not normal, then maybe we should be doing simulation for power of nonparametric tests such as Kruskal-Wallis. // Your summary is not wrong. If it helps match your intuition with reality, then good. But don't be surprised if others prefer somewhat different summaries. $\endgroup$
    – BruceET
    Dec 26, 2021 at 14:52
  • $\begingroup$ To clarify, I'm using ANOVA to compare results of gender (male, female, nonbinary) within my groups ie: expt group = 100, controls = 100. expt group So while my overall group numbers are equal, there is unequal number of male-female-nonbinary in my expt group for instance. And I'm trying to see if there are any gender differences in the results I'm getting from psych scales. $\endgroup$
    – user345213
    Dec 28, 2021 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.