I have three IVs with two levels/groups. My primary hypothesis is that there will be group differences for all IVs in total score. However, total score is comprised of scores on 7 subscales. Total score difference was hypothesised a priori, while subscale differences testing was done out of curiosity. 2x2x2 anova is out of the question because some groups n<5. 

How should I proceed with t-tests regarding the Bonferroni correction? My p is set at 0.01. 

It doesn't make sense to divide p with 24 (3x7+3x1) because the value is too low. Maybe for each IV p=0.01/8?

  • 1
    $\begingroup$ Can you clarify more the hypothesss you want to test? How many tests do you want to do? One per group (3)? Or for each group on each of the 7 outcomes? The argument that the test becomes too conservative is no excuse for choosing a more liberal cut off, it is the price you pay for using Bonferroni correction $\endgroup$ – Knarpie Sep 2 '17 at 12:07
  • $\begingroup$ I want to test group differences (age/sex/musicianship) on total DV score. For the first two I predict H0, for the last I expect a difference. However, I also want to check for differences on the 7 subscales of DV. Basically, for each group 8 hypotheses (1 for total score and 7 for subscales). $\endgroup$ – kapetantuka Sep 2 '17 at 12:46
  • $\begingroup$ And at which level do you want to control the FWER? Overall, or within each group of tests? And do you first test overall score and only then subscores if it is significant? Also note that your t-tests are not all independent. $\endgroup$ – Knarpie Sep 3 '17 at 9:16
  • $\begingroup$ My main research question are differences on total score depending on age, sex and musicianship. However, I also want to see if there are differences in subscales on the same variables because there could be differences in subscales that would not reflect on total score. Of course, subscale scores add to produce total score. I though it would be acceptable to set FWER at 1% for each separate analysis (p/8) instead of setting FWER for all the tests at 1% (p/24). There can be subscale differences irrespective of no difference in total score and I want to test them somehow. $\endgroup$ – kapetantuka Sep 3 '17 at 11:36
  • $\begingroup$ I understand you will do 24 tests in total, irrespective of the outcome of the tests for overall score. If you think that is acceptable, go ahead but remember the chance of making at least one type I error over all the 24 hypothesis is not guaranteed to be below 1% then. Dependence of tests is no problem according to Wikipedia: "This control does not require any assumptions about dependence among the p-values or about how many of the null hypotheses are true." $\endgroup$ – Knarpie Sep 4 '17 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.