4
$\begingroup$

I am doing a one-way ANOVA of a response variable (Y) on a treatment factor (T) of 7 levels. However, for each treatment, I have only 2 observations (or replicates). The ANOVA result shows the the means across the treatment are highly significant (p<0.001). I have check the normal and constant-variance assumptions and seems that my data does not violate these assumptions.

My question is: Can I trust the result of ANOVA, even if the sample size is extremely small (n=2*7=14)?

The first question that comes to my mind is the power of my test. Since my ANOVA has obtained an effect size of 0.9. I conducted a power analysis (using Q*Power.3) to see how many samples are needed to obtain such effect size with significance level of 0.05 and power of 0.8. The result showed that I need at least 28 samples. Since I have much fewer samples than required, does it mean that my ANOVA results are not reliable?

$\endgroup$
1
  • 3
    $\begingroup$ It really depends on what you mean by 'trust'. If the assumptions hold, the chance of a type I error is just your significance level - it's not a function of sample size (power sure is - but that's not an issue for you). Your power analysis shows that you don't have a high power ... but nevertheless you did reject. Small sample size doesn't increase the chance that you'll wrongly reject - on average it decreases it. $\endgroup$
    – Glen_b
    Mar 12 '13 at 0:12
1
$\begingroup$

It really depends on what you mean by 'trust'. If the assumptions hold, the chance of a type I error is just your significance level - it's not a function of sample size (power sure is - but that's not an issue for you). Your power analysis shows that you don't have a high power ... but nevertheless you did reject. Small sample size doesn't increase the chance that you'll wrongly reject - on average it decreases it.

$\endgroup$
1
  • 1
    $\begingroup$ I've copied this comment by @Glen_b as a community wiki answer because the comment is, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions. $\endgroup$
    – mkt
    Aug 24 '18 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.