1
$\begingroup$

I have a mixed design anova with 3 factors: drug (placebo vs. treatment), timing (pre, vs post-treatment), age group (young vs. elderly). The first two factors are within-subject and the third factor is between subject. I constructed the data frame and all the means look correct. There are total of 20 participants (10 young, 10 old). However the anova results are strange.

snippet of data

   part timing drug age freq_disc
1     1    pre  trt   Y  5.231433
2     2    pre  trt   Y  1.817267
3     3    pre  trt   Y  2.501767
4     4    pre  trt   Y  2.044167
5     5    pre  trt   Y  1.914400
6     6    pre  trt   Y  2.409567
7     7    pre  trt   Y  2.100533
8     8    pre  trt   Y  3.671033
9     9    pre  trt   Y  2.753967
10   10    pre  trt   Y  1.226233 

summary(anova_data)
      part     timing    drug    age      freq_disc      
 1      : 4   post:40   trt:40   O:40   Min.   : 0.9082  
 2      : 4   pre :40   plc:40   Y:40   1st Qu.: 1.7650  
 3      : 4                             Median : 2.4022  
 4      : 4                             Mean   : 3.8784  
 5      : 4                             3rd Qu.: 4.0412  
 6      : 4                             Max.   :24.6799  
 (Other):56                                             

anova command:

ez_an <- ezANOVA( data=anova_data, dv=freq_disc, wid=part, within=.(timing,drug), between=age ) 

$ANOVA
           Effect DFn DFd         F          p p<.05          ges
2             age   1  18 3.7628945 0.06823574       1.689433e-01
3          timing   1  18 3.3200863 0.08509702       2.556813e-03
5            drug   1  18 6.9669754 0.01665407     * 3.858303e-03
4      age:timing   1  18 0.1206849 0.73232183       9.316952e-05
6        age:drug   1  18 1.4761135 0.24008106       8.199634e-04
7     timing:drug   1  18 8.6809914 0.00863464     * 1.762140e-03
8 age:timing:drug   1  18 0.1939504 0.66489196       3.943761e-05

My question is: is it possible to have such significant effects of drug with those small effect sizes (ges column) given that the participants are only 20 total?

$\endgroup$
1
  • $\begingroup$ Power depends on $\Delta/\sigma,$ where $\Delta$ is size of difference to detect and σ is population standard deviation. So if σ (estimated by SD of appropriate residuals) is very small, then you can have small P-values for small actual differences. Will try to illustrate in Answer using t test $\endgroup$
    – BruceET
    Commented Jul 16, 2020 at 6:44

1 Answer 1

1
$\begingroup$

Here are two different scenarios that are alike, except that the smaller population standard deviation $\sigma$ in the second scenario gives high power for detecting the distance $\Delta = 1$ between two population means.

Scenario 1: Normal population 1 has $\mu_1 = 100,$ normal population 2 has $\mu_2 = 101,$ both populations have $\sigma = 5.$ Consider samples of size $n_1=n_2=10.$ In these circumstances a Welch two-sample t test rejects at the 5% level only about 7% of the time. So it will be rare to find a significant difference when the real difference is 1.

set.seed(715)
pv = replicate(10^5, t.test(rnorm(10,100,5),rnorm(10,101,5))$p.val)
mean(pv <= .05)
[1] 0.06871

A typical two-sample dataset in this scenario can be summarized as follows. It is one of the many such datasets in which no significant difference is found.

set.seed(715)
x1 = rnorm(10, 100, 5)
summary(x1);  sd(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  91.74   99.73   99.96   99.92  101.39  106.41 
[1] 3.665256  ## SD for Sample 1
x2 = rnorm(10, 101, 5)
summary(x2);  sd(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  90.97  100.43  101.77  102.09  106.09  110.95 
[1] 5.801608  ## SD for Sample 2

boxplot(x1,x2, horizontal=T, col="skyblue2", pch=20)

enter image description here

t.test(x1, x2)

         Welch Two Sample t-test

data:  x1 and x2
t = -0.99765, df = 15.197, p-value = 0.3341
alternative hypothesis: 
  true difference in means is not equal to 0
95 percent confidence interval:
 -6.785208  2.455214
sample estimates:
mean of x mean of y 
 99.92492 102.08991 

Scenario 2: Normal population 1 has $\mu_1 = 100,$ normal population 2 has $\mu_2 = 101,$ both populations have the smaller standard deviation $\sigma = 0.5.$ Again, consider samples of size $n_1=n_2=10.$ In these circumstances a Welch two-sample t test rejects at the 5% level over 98% of the time. So one will almost always find a significant difference when the real difference is 1.

set.seed(716)
pv = replicate(10^5, t.test(rnorm(10,100,.5),rnorm(10,101,.5))$p.val)
mean(pv <= .05)
[1] 0.98746

A typical two-sample dataset in this scenario can be summarized as follows. It is one of the many such datasets in which a significant difference is found.

y1 = rnorm(10, 100, .5)    
summary(y1);  sd(y1)
    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   99.41  100.07  100.25  100.22  100.40  100.89 
[1]  0.4248364
y2 = rnorm(10, 101, .5)
summary(y2);  sd(y2)    
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  100.2   100.8   101.0   101.0   101.3   101.6 
[1] 0.4327343

This boxplot uses about the same horizontal scale as the previous one.

boxplot(y1,y2, horizontal=T, col="skyblue2", pch=20, ylim=c(91,111))

enter image description here

t.test(y1, y2)

        Welch Two Sample t-test

data:  y1 and y2
t = -3.9775, df = 17.994, p-value = 0.0008835
alternative hypothesis: 
  true difference in means is not equal to 0
95 percent confidence interval:
 -1.1656442 -0.3598506
sample estimates:
mean of x mean of y 
 100.2202  100.9830 
$\endgroup$
1
  • $\begingroup$ thanks for the nice explanation $\endgroup$
    – abc
    Commented Sep 25, 2022 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.