0
$\begingroup$

I have a mixed design anova with 3 factors: drug (placebo vs. treatment), timing (pre, vs post-treatment), age group (young vs. elderly). The first two factors are within-subject and the third factor is between subject. I constructed the data frame and all the means look correct. There are total of 20 participants (10 young, 10 old). However the anova results are strange.

snippet of data

   part timing drug age freq_disc
1     1    pre  trt   Y  5.231433
2     2    pre  trt   Y  1.817267
3     3    pre  trt   Y  2.501767
4     4    pre  trt   Y  2.044167
5     5    pre  trt   Y  1.914400
6     6    pre  trt   Y  2.409567
7     7    pre  trt   Y  2.100533
8     8    pre  trt   Y  3.671033
9     9    pre  trt   Y  2.753967
10   10    pre  trt   Y  1.226233 

summary(anova_data)
      part     timing    drug    age      freq_disc      
 1      : 4   post:40   trt:40   O:40   Min.   : 0.9082  
 2      : 4   pre :40   plc:40   Y:40   1st Qu.: 1.7650  
 3      : 4                             Median : 2.4022  
 4      : 4                             Mean   : 3.8784  
 5      : 4                             3rd Qu.: 4.0412  
 6      : 4                             Max.   :24.6799  
 (Other):56

anova command:

ez_an <- ezANOVA( data=anova_data, dv=freq_disc, wid=part, within=.(timing,drug), between=age ) 

$ANOVA
           Effect DFn DFd         F          p p<.05          ges
2             age   1  18 3.7628945 0.06823574       1.689433e-01
3          timing   1  18 3.3200863 0.08509702       2.556813e-03
5            drug   1  18 6.9669754 0.01665407     * 3.858303e-03
4      age:timing   1  18 0.1206849 0.73232183       9.316952e-05
6        age:drug   1  18 1.4761135 0.24008106       8.199634e-04
7     timing:drug   1  18 8.6809914 0.00863464     * 1.762140e-03
8 age:timing:drug   1  18 0.1939504 0.66489196       3.943761e-05

My question is: is it possible to have such significant effects of drug with those small effect sizes (ges column)?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

I only have a few college stat courses under my belt so maybe not an expert, I agree you may have better luck elsewhere, but yes I believe you can have very significant results even with small effects, as long as the sample size is large.

The p value given is about the probability that the effect is 0. If you ran a billion tests and every time the effect was measured between 0.001 and 0.0001, you could be very sure the answer was greater than zero, even if only slightly. So you can be confident in small effects.

You can learn more about p-values in general and what they mean here: https://towardsdatascience.com/p-values-explained-by-data-scientist-f40a746cfc8

EDIT: check out dbwilson's comment below for suggestions of what to do here, since n is small

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ This answer is correct. However, in this situation the sample size is small (n=20). I'm not sure how ezANOVA is computing Hedges' g for the within-subjects effects -- this is not as straightforward as it may seem. Focus on the raw mean differences. Are they big? Also, try using R's "aov" function to see if you get the same results. $\endgroup$
    – dbwilson
    Commented Jul 16, 2020 at 12:45
  • 1
    $\begingroup$ Oh you're totally right, I totally skimmed over the 20 people part. I clearly should not be trying to help at 2 am lol $\endgroup$
    – Randcelot
    Commented Jul 16, 2020 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.