Hochberg/Bonferonni correction for multiple one-sample t test I am trying to conduct a content validity analysis where I want to test if a certain item (e.g. a question) can be related to a certain construct (e.g. depression). I've let people fill in a questionnaire, asking them if they would assign an item to a construct and with how much certainty (0 - 100%). I've got my data, did the processing and executed a one-sample t-test for every combination item - construct.
For example I test, does the participants average value for item 1 related to construct 1, significantly differ from 0 (there is a positive linkage). I do this for every item & construct combination. I have 20 items & 7 constructs; thus execute 140 t-tests.
A paper suggested correcting my p-values with a bonferroni or hochberg correction because I've performed multiple tests.
From what I understand a correction looks at an amount of tests to reduce the 'familywise error rate'. My question is, what is a family in my case? Is this 20 t.tests within a construct family, 7 t.tests within an item family or 140 t.tests within my combination of construct-item?
 A: The definition of "Family" depends on the situation, and in some cases (yours) is somewhat ambiguous. The most important thing is to state what you did clearly. Since you didn't plan the analyses in advance, probably the fairest approach would be to use the most conservative approach, setting the family to all 140 t-tests.
A: There is no simple "true" or "false" regarding whether and how p-values should be adjusted for multiple testing. Depending on how exactly do it, you have different mathematical guarantees; meaning that you can achieve certain maximum error probabilities over whatever you define as "family", at the price of a reduced power (the more severe the correction, the weaker the power; Bonferroni over all 140 is the strictest of them all).
The most important thing about multiple testing correction in my view is to understand that and how interpretations of results change depending on how you do it. The more severe the correction, the more reliable your significances, but the higher the chance to miss things that are going on. It's a trade-off.
Here's something to consider. Tests are often criticised for being interpreted in a binary accept/reject fashion, which oversimplifies the situation and can give a false sense of certainty. You have the option to use various rules (Bonferroni and Benjamini-Hochberg in families of different size) and state for each p-value at the level from what rule it is significant. Obviously the more severe the rule, the more reliable this significance, and you could go down to uncorrected significance and say that these may be of interest for further investigation but could easily be meaningless due to multiple testing.
