0
$\begingroup$

I'm a grad student doing analysis on a survey. One of the things I want to note from this study is whether demographic factors of respondents are connected to the answers they gave (how do men or women think about X, do different age cohorts think differently about Z, etc).

I've done chi squared tests of independence looking at 10 demographic factors and the answers to 10 questions (100 total tests). I want to apply a Bonferroni correction for multiple comparisons. However, I am not sure if I am supposed to apply a correction factor of 10 (each question had 10 demographic factors tested to see if any of them were significant), or if I need to apply a correction factor of 100, since I did 100 total comparisons across all questions. (Either way, I still have lots of significant results.)

Thanks!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If you were going to potentially claim that any of these 100 would be a significant finding as the main finding of a paper (and are looking at nothing else as a potential main finding), then 100. With 100 tests, it will usually be a good idea to use something less conservative than Bonferroni (e.g. Bonferroni-Holm). Another consideration is that you do not really seem to do confirmatory hypothesis testing, because you are still defining the exact analysis method after already having seen the data, which quite honestly opens up the whole garden of forking paths anyway.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks! I tried using both the Bonferroni and Holm methods in R, but they gave me the same amount of correction, such as changing a 0.002301 to 0.2301. Do the correction methods not diverge for large numbers of comparisons? (Example R code: p.adjust(0.002301, method = "bonferroni", n = 100) results in 0.2301 for either method.) $\endgroup$
    – aeluro
    Apr 1, 2019 at 17:42
  • $\begingroup$ It will depend on what happened on the other comparisons. Once at least one adjusted p-values was significant, the other adjusted ones should change. $\endgroup$
    – Björn
    Apr 1, 2019 at 20:22
  • $\begingroup$ Thank you. Since that's not what I'm getting using the standard way of adjusting p-values in R, do you happen to know another way of applying a Bonferroni-Holm correction using R? $\endgroup$
    – aeluro
    Apr 2, 2019 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.