I am comparing biomarker levels (50 total) obtained from immunoassay in a cohort divided into 3 groups (control vs. inactive vs. active).

I want to use the Kruskal-Wallis test with Dunn’s test to assess if there are differences between the groups (control vs. inactive, control vs. active, inactive vs. active).

I would then like to calculate a q-value to estimate the false discovery rate (using the p.adjust function in R) given the multiple comparisons. I’ve done this before comparing 2 groups (control vs. disease) with the Mann-Whitney test, and input all the p values to yield the adjusted p values (or q values).

When doing the same thing after the Kruskal-Wallis test, I am confused as to when to adjust the p’s. Is it after performing KW w/Dunns on all 50 biomarkers and obtaining a final p value, then inputting them into the p.adjust code? Would I only adjust the ones that yielded significance on the initial Kruskal-Wallis? Would it be better to compare the groups individually with the Mann-Whitney test (control vs. inactive, control vs. active, inactive vs. active) and then adjusting all 50 p values?


1 Answer 1


The issue here is an extension of the general question about whether omnibus tests like Kruskal-Wallis need to be done or don't need to be done before pairwise comparisons. This page and its links provide a lot of food for thought on this matter.

The answer depends on what you are trying to accomplish with your study.

If you are trying to identify promising biomarkers that are associated with the 3 groups overall, then it would make sense to do the 50 Kruskal-Wallis tests and adjust those 50 p values to identify the top candidate biomarkers. That would seem to be the most fruitful for guiding further studies. Then perhaps focus on biomarkers that most clearly distinguish all 3 groups.

At the other extreme, if you are trying to identify the biomarkers most associated with any of the 3 pairwise comparisons within each of the groups, then use the p values for all 150 potential pairwise comparisons (3 per biomarker, 50 biomarkers). There's nothing wrong with that in principle, but it's not clear how useful that would be for guiding further studies.

For example, do you really want to be in a situation where you say something like: "evaluate marker A for the control vs. inactive difference, marker B for the control vs. active difference, and marker C for the inactive vs. active difference"? That's what you might end up with if you just evaluate all 150 pairwise comparisons.

In general, it is OK to do multiple-comparison correction only for a limited pre-planned set of pairwise comparisons even if many more pairwise comparisons might be possible. If you pre-planned only to evaluate pairwise comparisons after a significant Kruskal-Wallis test on a biomarker, that would mean "only adjust[ing] the ones that yielded significance on the initial Kruskal-Wallis." Or if all you care about are differences of the other 2 groups from control, you could limit your pairwise comparisons to those 2 differences.

One potential complication in your design is that you presumably are measuring all 50 biomarkers within each subject. Nothing in your design accounts for the correlations of those biomarker levels within each individual.

Consider whether you really needed to do Kruskal-Wallis instead of a method that might better accommodate the handling of intra-subject correlations. Standard linear modeling of biomarker levels (perhaps after log or some other transformation) is one possibility; for inference you don't need normal distributions of observed values, only a situation in which the sampling distribution of the regression coefficients is close enough to normal. Or recognize that Kruskal-Wallis is just a special case of ordinal regression, and use that more generally applicable method instead.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.