I gave three groups of participants (say groups A, B, and C) different treatments, then gave all groups the same questionnaire. The questionnaire has ~50 questions. I then run ANOVA on each of the questions to see if there are differences between the groups. Is this a common practice? Is it bad to be doing this many ANOVA tests?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It may be different from your situation, but I had a similar problem in an application with drug treatments. In particular, I was asked to test via ANOVA with repeated measures if a treatment was doing something different with respect to other, more standard, treatments on six different biological variables (such as IGG, TNFa, IL10 etc.). I knew that I had to control for multiplicity within variable (e.g. for IGG, TNFa, IL10 etc.), but was unsure if a correction was necessary also for comparisons between variables.
A (statistician) friend of mine, much more expert than me on multiple testing issues, suggested me to also control for the second issue. Hence I had to do a double adjustment for multiplicity, this killed a lot o pvalues but, at least, in my case, the procedure was statistically sound. Hope this can help.
