1
$\begingroup$

I am trying to conduct a content validity analysis where I want to test if a certain item (e.g. a question) can be related to a certain construct (e.g. depression). I've let people fill in a questionnaire, asking them if they would assign an item to a construct and with how much certainty (0 - 100%). I've got my data, did the processing and executed a one-sample t-test for every combination item - construct.

For example I test, does the participants average value for item 1 related to construct 1, significantly differ from 0 (there is a positive linkage). I do this for every item & construct combination. I have 20 items & 7 constructs; thus execute 140 t-tests.

A paper suggested correcting my p-values with a bonferroni or hochberg correction because I've performed multiple tests.

From what I understand a correction looks at an amount of tests to reduce the 'familywise error rate'. My question is, what is a family in my case? Is this 20 t.tests within a construct family, 7 t.tests within an item family or 140 t.tests within my combination of construct-item?

$\endgroup$
3
  • $\begingroup$ I'm a bit confused by your explanation. For sure you don't mean that the participant average is an average of the certainty values (as these are >=0), but then I'm not quite sure what the values are on which you run your tests, and how exactly they are combined with the constructs to define these tests. $\endgroup$ Commented Jul 31, 2022 at 21:28
  • $\begingroup$ Every participant had to answer 2 questions for every combination item - construct. 1) Does this item belong to this construct? (Yes or No) 2) How confident are you in your statement (0 - 100%) $\endgroup$ Commented Aug 1, 2022 at 7:58
  • $\begingroup$ I still don't understand what is tested to "differ significantly from zero". $\endgroup$ Commented Aug 1, 2022 at 15:05

2 Answers 2

1
$\begingroup$

The definition of "Family" depends on the situation, and in some cases (yours) is somewhat ambiguous. The most important thing is to state what you did clearly. Since you didn't plan the analyses in advance, probably the fairest approach would be to use the most conservative approach, setting the family to all 140 t-tests.

$\endgroup$
0
$\begingroup$

There is no simple "true" or "false" regarding whether and how p-values should be adjusted for multiple testing. Depending on how exactly do it, you have different mathematical guarantees; meaning that you can achieve certain maximum error probabilities over whatever you define as "family", at the price of a reduced power (the more severe the correction, the weaker the power; Bonferroni over all 140 is the strictest of them all).

The most important thing about multiple testing correction in my view is to understand that and how interpretations of results change depending on how you do it. The more severe the correction, the more reliable your significances, but the higher the chance to miss things that are going on. It's a trade-off.

Here's something to consider. Tests are often criticised for being interpreted in a binary accept/reject fashion, which oversimplifies the situation and can give a false sense of certainty. You have the option to use various rules (Bonferroni and Benjamini-Hochberg in families of different size) and state for each p-value at the level from what rule it is significant. Obviously the more severe the rule, the more reliable this significance, and you could go down to uncorrected significance and say that these may be of interest for further investigation but could easily be meaningless due to multiple testing.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.