0
$\begingroup$

I want to validate a questionnaire with three scales summing up to a total score. For for all tests I conducted, I tested the scales separately and additionally the total score (so 3+1 and that total one bearing all the summed up information of the first 3 scales). I've tested:

  • Skewness and Kurtosis of the scales to find out about deviation of normal distribution and Cronbach's Alpha for all scales
  • An ANOVA to test gender differences
  • independent samples t-tests for each scale (3+1) to test for group differences between psychology students and student of other fields of study
  • Several product-moment-correlations with the scores of three further questionnaires measuring convergent or similar concepts and also sometimes having more that one scale (4/2/1)

So, I have a lot of p-values in the scope of the study.

What I don't understand:

  • Do I really have to divide the Alpha level by the number of all the tests conducted (4*Skewness + 4*Kurtosis + 4*Alpha ( + even the alphas of all the other scales --> 4+2+1) + ANOVA (for all scales 4+4+2+1) plus 4*t-tests + (4*4 + 4*2 + 4*1) Product-moment-correlations. Or do I calculate one Alpha level for each scale in dependence of how often this scale was involved in any testing?
  • Then I don't now how to handle the fact, that the fourth scale (the total score) consists of all the items of the three "subscales".
Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

You have a bunch of dependent tests. The advantage of Bonferroni is that it doesn't care about the dependence. The price is a low power because you have to divide $\alpha$ by the total number of tests even though some tests are very related.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for your answer, Nik. Does that mean, that in general the probability to find random significant results decreases if the tests run on the same data set are related? Do you have any recommendation what to do in my case? $\endgroup$
    – Luise
    Commented Jul 19, 2016 at 10:47
  • $\begingroup$ It depends on how exactly the tests statistics are related. E.g. if there are two tests that their t-statistics can have positive or negative correlation, and if you know what it is or how to estimate it, you can increase the power. In your case it's not just one correlation coefficient but a large correlation matrix whose size is equal to the total number of tests. The advantage of Bonferroni is that it works for any correlation matrix. Why don't you first try using Bonferroni - perhaps you already have enough power despite a large number of tests. $\endgroup$
    – Nik Tuzov
    Commented Jul 19, 2016 at 14:33
  • $\begingroup$ Of course the advantage of Benjamini & Hochberg's false discovery rate correction is that it does not hemorrhage statistical power like Dunn's Bonferroni correction, does not depend on a formally undefined notion of "family of tests", and does not depend on an assumption that nothing one studies is related to anything else, even in the face of mounting evidence. Also, it only cares about negative dependence of rejection probabilities. Plus it's about four decades more up to date. :) $\endgroup$
    – Alexis
    Commented Feb 5, 2019 at 15:59
-1
$\begingroup$

How exactly are you using the p-values to draw a conclusion? That can make a big difference in what the appropriate adjustment for multiple comparisons is. For example, if you are interested in demonstrating significance (e.g., demonstrating that groups score differently on the scale), then such tests should probably be adjusted, as the chance of a false positive increases with every test you do. On the other hand, if you are trying to demonstrate NONsignificance (e.g., confirming an absence of evidence that groups differ on the scale) then it is probably not appropriate to adjust such tests, as adjusting would inflate--rather than reduce--your chance of demonstrating what you want.

For assumption tests (such as for skewness and kurtosis), generally you are trying to show a nonsignificant p-value, not a significant p-value, so it's rarely appropriate to include those in your adjustments.

Improve this answer
$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.