2
$\begingroup$

We are investigating the relationship between smoking status and the hormone level. Smoking status is X(three-level) and the level of the hormones is Y(continuous variable). However, the smoking status consists of three levels in our sample and there are 11 different types of hormone. I saw my friends did the repeated Kruskal-Wallis test (which is the non-parametric parallel to the ANOVA) for different hormone but make the adjustment to p-value using Benjamini-Hochberg procedure and set the number to 11. He explains to me that there are 11 different hormones but to my understanding, Benjaminin-Hochberg procedure is used to adjust for the multiple comparisons of three different smoking status.

I am not sure who is correct? Would it be valid to set Benjamini-Hochberg procedure to make adjustment assuming there are 11 different levels to compare? If not, what should we do to tackle the multivariate problems using non-parametric method (because the hormone level is not normally distributed)?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ If you are conducting exploratory or hypothesis generating research then you should not adjust the p-values for multiplicity. And, quite likely, you should not adjust even if your research is not exploratory. Search this site for previous discussions of adjustment for multiplicity. $\endgroup$
    – Michael Lew
    Commented Sep 29, 2019 at 20:32
  • $\begingroup$ @MichaelLew I have checked the adjustment for multiplicity but don't know which one specifically should I look into. Also, when you say we shouldn't adjust the p-value, do you mean by using ANOVA for each hormone, we shouldn't make the adjustment to the p-value obtained from the chi-square statistics? Because we can also adjust the p-value of ad hoc test(Dunn's test) for each KW test. $\endgroup$
    – JoZ
    Commented Sep 30, 2019 at 15:34

1 Answer 1

Reset to default
1
$\begingroup$

One hormone. For each hormone, a one-way (one-factor) ANOVA or Kruskal-Wallis test will have three levels of the factor (groups for smoking status). First, you might do a test at the 5% level to see if there are any differences in hormone assays among the three groups. If that overall F-test shows no differences, you should not investigate further to find differences among groups. (Such further investigation would run an unusually high risk of 'false discovery'; you've already been "told" there are no legitimately significant differences.)

If the overall null hypothesis is rejected, suggesting that there are inter-group differences, you would typically do 'ad hoc' tests to determine the pattern of differences. A vs B, B vs C, and A vs C. You should use some method to avoid 'false discovery' in declaring differences among groups. One method (out of maybe a dozen in general use) is the Bonferroni method ,which says these three 'ad hoc' tests should be done at level 5%/3 = 1.7% level.

All 11 hormones. If you are doing individual ANOVAs for each of 11 hormones, you need to be aware that testing each at the 5% level, you will have more than a 5% chance of falsely finding effects somewhere among the 11 ANOVAs--even if there are no homormonal changes at all due to differences in smoking.

Opinions differ as to how to mitigate this risk. From what you say in your question, I suppose you are planning to use the Benjamini-Hochberg procedure in this regard (one of several discussed in the Wikipedia link--near the end). It might be appropriate to do a multivariate ANOVA in which all 11 hormones are tested simultaneously and the protection against false discovery is built into one overall test.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ The idea of 'rejecting' 'hypotheses' is coming under fire, for good reasons. Regardless of the result of a hypothesis test it is a better strategy to estimate differences (point + interval estimates; even better the whole Bayesian posterior distribution). For rank tests this can be done with the proportional odds ordinal logistic model, which generalizes the K-W test. $\endgroup$
    – Frank Harrell
    Commented Sep 30, 2019 at 11:19
  • $\begingroup$ Thank you so much for your answer and also thanks @FrankHarrell for the useful add on. We did the normality test and rejected the normality assumption first. Therefore we only conducted the K-W test but only for each individual hormone. The R return each one hormone a p-value, and what our group did is to apply BH procedure directly to the p-value returned. Then we select the groups that are still significant after adjustment and do the ad hoc test (Dunn's test) to furthur investigate the inter-group differences. $\endgroup$
    – JoZ
    Commented Sep 30, 2019 at 14:32
  • $\begingroup$ If I understand the answer correctly, it seems the way we apply the BH procedure is not correct. Are their any non-parametric parallel to MANOVA that we can utilize and make the adjustment using BH method? Also, after selecting the significant groups, does that mean either we can do a ad hoc test (Dunn test for example) or we can follow what @FrankHarrell suggest, applying proportional odds ordinal logistic model to each individual hormones? Mank thanks for your answers. $\endgroup$
    – JoZ
    Commented Sep 30, 2019 at 14:43
  • 1
    $\begingroup$ You should not use any formal tests for normality; they are typically overly-sensitive, while ANOVAs are very robust to deviations from normality. Moreover, you can rely on the central limit theorem as far as inferences of 'significant' effects go. MANOVAs are similarly robust, so I would not worry about looking for non-parametric alternatives. $\endgroup$
    – André.B
    Commented Sep 30, 2019 at 20:39
  • $\begingroup$ The central limit theorem does not work well enough for that to hold. This is especially true for skewed distributions for which the standard deviation is not even a valid measure of dispersion. $\endgroup$
    – Frank Harrell
    Commented Oct 1, 2019 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.