# Power Analysis for a mixed two-way ANOVA with unqual sample sizes

I have a simple study design with a between-subject factor drug (real drug vs. placebo) and a within-subject factor time (three visits= repeated measures). I want to run a mixed two-way ANOVA and my hypothesis is that peoples health improves more if they receive a real drug.

I need to calculate the sample size with the following parameters: alpha= 0.05, power= 0.90, effect size f= 0.125 and a correlation bewtween the repeated measures at the visits of r= 0.62. In addition, I need to take into account unequal sample sizes. For example, what sample size is needed if I accept 10% more patients in one group than in the other? So far I could not find how to calculate power for an unbalanced mixed ANOVA, neither with G*Power nor with R. Thanks in advance.

• Maybe you should try the simulation. Oct 8, 2018 at 13:40
• If you can provide equations for Cohen's f or alternative effect size, I can demonstrate how you could program a solution in R. Oct 9, 2018 at 0:00
• @ a_statistician: could you specify the simulation method you are suggesting?
– user213325
Oct 9, 2018 at 8:34
• @ Heteroskedastic Jim: To my knowledge there are different ways one can calculate Cohen's f, for example from eta-squared (www-01.ibm.com/support/docview.wss?uid=swg21476421) and eta-squared can be calculated from the Sum of Squares (hosted.jalt.org/test/bro_28.htm) and so on. But to me it is unclear why this is relevant since I have already provided a Cohen's f value (f= 0.125).
– user213325
Oct 9, 2018 at 8:36
• @IgoR I asked because it helps to be able to translate the model into a regression. I typed up an answer some days back assuming f = root of f^2 and will post it now. Oct 12, 2018 at 0:08

The simulation method to calculate the sample size when the statistical methods is not simple.

1. Write the statistical model mathematically.

2. Generate the sample according to the model with sample size N.

3. Fit the model, perform the test, and record the rejection or acceptance of hull hypothesis.

4. Repeat step 2 and 3 n (generally I used 5000) times.

5. Calculate the power by (# of rejections)/n.

6. If power is too higher, decrease sample size N, repeat 2 - 5. If power is too lower, increase sample size N, repeat 2 - 5.

Given the current computer speed, after you finish the programming, it will not take days to find the suitable sample size.

If the method of statistical test is relatively new, it is better to generate the data under the null hypothesis, such that you can verify the alpha level of the test.

• To make sure I get it right: Do I consider the unequel sample sizes in step two, where I generate the data?
– user213325
Oct 25, 2018 at 6:04
• You make the decision on sample size in each group. Any values are acceptable in this procedure. Oct 25, 2018 at 16:49

The first thing I can't figure out is what Cohen's $$f$$ is. I'll assume it's the square root of Cohen's $$f^2$$? It makes sense to make this into a regression, and then it's easier to work with variance explained like $$\eta^2$$ and $$R^2$$. According to Wikipedia, Cohen's $$f^2$$ is: $$f^2=\frac{R^2}{1-R^2}$$ so $$R^2=\frac{f^2}{f^2+1}=\frac{0.125^2}{1+0.125^2}=0.015$$ So that's one component.

Next, we can also figure out the correlations between the repeated measures. If $$\rho=0.62$$, we can first generate three columns of data from a multivariate normal distribution:

set.seed(12345)
n <- 200000
cor.mat <- matrix(.62, 3, 3)
diag(cor.mat) <- 1
dat <- as.data.frame(MASS::mvrnorm(n, rep(0, 3), cor.mat))
cov(dat) # Variances should be about 1, and covariances about .62

V1        V2        V3
V1 0.9999592 0.6187661 0.6203490
V2 0.6187661 1.0040113 0.6228203
V3 0.6203490 0.6228203 1.0043763


To simplify things, I'll assume the $$f^2$$ comes any one of the three time points. If I do that, then I have the following equation: $$y=\beta\times treat+\epsilon$$ where $$y$$ is the data from any one time point, and for now $$\epsilon$$ is any column from the data frame above. Nicely, it has a variance of 1 like the columns of the data frame we just generated. If $$f=0.125$$, then $$0.125^2=\mathrm{Var}(\beta\times treat)=\beta^2\times\mathrm{Var}(treat)$$

Since you specified that one group has proportions 10% greater than the other, I'll assume a 40-60 split, with 40% in the treatment group (it could readily be the other way round). This is equivalent to $$treat\sim Bern(0.4)$$. And since $$treat$$ is Bernoulli, its variance is $$0.4 \times 0.6=0.24$$. So for $$0.125^2=\beta^2\times\mathrm{Var}(treat)$$, $$\beta=\frac{0.125}{sd(treat)}=\frac{0.125}{\sqrt{(0.24)}}=\frac{0.0625}{\sqrt{0.06}}$$

dat$$treat <- rbinom(n, 1, .4) mean(dat$$treat); var(dat$treat) [1] 0.39928 [1] 0.2398567 beta <- 0.0625 / sqrt(0.06) dat$$y1 <- beta * dat$$treat + dat$$V1 dat$$y2 <- beta * dat$$treat + dat$$V2 dat$$y3 <- beta * dat$$treat + dat$V3


A quick check on things:

c(summary(lm(y1 ~ treat, dat))$$r.squared, summary(lm(y2 ~ treat, dat))$$r.squared,
summary(lm(y3 ~ treat, dat))$r.squared) [1] 0.01576065 0.01593496 0.01562384  Seems about right. Because the groups are unbalanced, you should analyze using lmer()/lme() from lme4/nlme as suggested in the anova() documentation. I'll use lmerTest so I can obtain p-values. First, I need to reshape the data to long format: dat$ID <- 1:nrow(dat)
dat <- reshape(
dat, idvar = "ID", varying = list(c("y1", "y2", "y3")), direction = "long")


Then:

library(lmerTest)
res <- lmer(y1 ~ treat + (1 | ID), dat)
coef(summary(res))["treat", 5] # this is what we want
[1] 0


Now we can replicate this process at different sample sizes to get a sense of the power:

# Essential components
cor.mat <- matrix(.62, 3, 3)
diag(cor.mat) <- 1
beta <- 0.0625 / sqrt(0.06)
alpha <- .05

res <- t(replicate(1000, {
n <- sample(seq(50, 500, 50), 1)
dat <- as.data.frame(MASS::mvrnorm(n, rep(0, 3), cor.mat))
dat$$treat <- rbinom(n, 1, .4) dat$$ID <- 1:nrow(dat)
dat <- reshape(
dat, idvar = "ID", varying = list(c("V1", "V2", "V3")), direction = "long")
dat$$y <- dat$$V1 + beta * dat\$treat
res <- lmer(y ~ treat + (1 | ID), dat)
c(n, coef(summary(res))["treat", 5] < alpha)
}))
colnames(res) <- c("n", "power")
res.agg <- aggregate(power ~ n, res, mean)
plot(power ~ n, res.agg)


These are the barebones of the kind of power analysis you can conduct. With this, you have a general sense of the sample sizes you need to be at for 90% power; it's a number above 300. You can further narrow down the sample size in the power analysis.

Is the Cohen's $$f$$ computed using an average of the three scores, how was this between effect generated specifically? That can affect data generation and the procedure above. Also, if you use lmerTest, you should explore the options for obtaining p-values.