0
$\begingroup$

Quick question regarding adjustung p-values....

I preformed a logistic mixed effects regression model and used Wald likelihood ratio test to select model parameters.

I received comments back from a journal reviewer stating that I should use Holm-Bonferroni correction for p-values due to multiple comparisons....

I did not think that adjusted p-values were required when performing mixed effects logistic regression.... am I incorrect in my assumption? any comments/suggestions greatly appreciated, thanks

$\endgroup$
  • $\begingroup$ I think you either used the Wald test or the likelihood ratio test, they are not the same. If I were reviewing this for a journal I would be more worried about your having performed variable selection than any issue of multiple comparisons. $\endgroup$ – mdewey Jun 2 '16 at 15:39
1
$\begingroup$

Using a Wald likelihood ratio test to select model parameters will invalidate any p-values and confidence intervals, as well as biasing estimates - unless you adjusted for that somehow. Thus, the reviewer is right to highlight this topic, if you had not done some appropriate there - especially so, if the model building is an important part of the paper. Just reporting estimate, CI and p-value after model selection as if not model selection had occured is wrong and misleading.

I am not sure that using a Bonferroni-Holm correction is an approach for dealing with that. Perhaps it is possible to use it for that purpose at least to correct the p-values, but it may be extremely conservative.

If you then afterwards are performing multiple comparisons across multiple levels of a random effect, then there's some people that would argue that no correction is needed (see e.g. Gelman et al.'s "Why we (usually) don't have to worry about multiple comparisons" article) - but others (perhaps including your reviewer) may disagree. If you are comparing multiple levels of some fixed effects factor, then it is clearer that there is a multiple comparison issue (although some might still argue against the need for type I error correction if this is rather exploratory and you do not want to use a hypothesis testing approach claiming "statistical significance").

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Multiple testing corrections are not inherently linked to an particular analytic tool, just to hypothesis testing in general.

p.s. I would suggest using something less conservative than the bonferroni correction, especially if you have A LOT of tests. For example, look into controlling the false discovery rate, rather than the family-wise error rate.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.