2
$\begingroup$

I am using linear mixed effect models to analyse my data. I am looking at how 4 independent variables might affect 5 dependent variables. I read an article by von der Malsburg and Angele (2017) which raises the concern about false positives and suggests that Bonferroni Correction is an appropriate remedy. I then found an online software that can help to calculate the adjusted t and p value after Bonferroni Correction (https://www.quantitativeskills.com/sisa/calculations/bonfer.php?Alpha=0.05&N=9&Corr=0.6&Df=00). However, I am not sure about the number of test that I should put there. Should it be the number of dependent variable, which is 5? In other words, the adjusted alpha is 0.05/5=0.01?

Another related issue is that, if I opt for Bonferroni Correction by using alpha of 0.01 with t value of 2.37 (two-tailed), then I don't need to use any p-adjusted correction when doing pairwise comparison with emmeans function? In other words, is it safe to use the following code for pairwise comparison (A is an independent variable of 3 factors), and then use the t value threshold of 2.37 and p of 0.01 to determine whether there is any significant effect in these comparisons:

emm_ <- emmeans(lmerModel, "A")

contrast(emm_, "pairwise", adjust = "none")

$\endgroup$
1
$\begingroup$

I honestly can't tell how many tests are contemplated or what is the right course of action.

First the technical question: The idea of coding for no adjustment, but later using an altered threshold for the P values is always a workable option when applying the Bonferroni correction, because all you have to do is divide your required threshold (say, 0.05 -- which people very, very seldom give any serious thought) by the number of tests. However, if you are dealing with an audience that is accustomed to basing their decisions on adjusted P values, that approach is highly problematic because as soon as people see numbers, their attention is immediately diverted from whatever cautionary words accompany them.

So, we have 5 responses and 4 independent variables. Some of those independent variables are factors. The number of tests performed depends on the numbers of levels of the factors. And there is the possibility that some of them interact, leading to more detailed comparisons, stratified by levels of other factors. So who knows how many tests are to be performed in total?

And is it truly advisable to view the whole shootin' match as a single family of tests, for which we want to control the error rate? That is a question that should be addressed in the context of the subject matter. And how tightly should the error rate be controlled? The probability of making any type-I error, as is done with Bonferroni? Or something less stringent such as the false discovery rate? These kinds of questions are actually no much unlike our current controversies about face masks: How much protection is reasonable, and over what group?

All that said, I very rarely have seen people apply the concept of a "family of tests" beyond one dependent variable. I usually see people consider a "family" defined as comparisons of levels of one factor, for one dependent variable. And even when other factors are involved, I think I most often see families of comparisons conditioned on the levels of other factors, each separately adjusted. That is what seems the most common practice. It may or may not be the best practice. And it depends too on those separate dependent variables; are they different measures of the same thing? Or completely different?

But it is clear that it is possible to go too far in protecting an overall error rate. So I wouldn't go too far overboard, because you'll find yourself serving some priggish external objectives rather than reporting useful findings. And remember that no research should be the final say on a particular topic. So your goal is to avoid over-stating the importance of your results, but leave room to invite others to consider what may be most important and worthy of future investigation.

$\endgroup$
2
  • $\begingroup$ Thank you for your detailed explanation. Another related question: if an independent variable (A) has 3 factor levels (a, b, c) , and I use sliding contrast in the lmer model to compare consecutive levels (i.e., a vs. b, c vs. b), then the model summary will give me the results of the differences between a and b, b and c,is there any default correction method used by lmer when doing these multiple comparisons and producing the model summary? $\endgroup$ – Chloe Jun 8 at 13:29
  • 1
    $\begingroup$ lmer doesn't output adjusted P values. The "consec" method for emm::contrast() is available for such contrasts. $\endgroup$ – Russ Lenth Jun 8 at 14:04
1
$\begingroup$

Since you are making $5\cdot 4=20$ tests, the adjusted alpha should be $0.05/20=0.0025$ in the first part where you are finding significant relations.

In the second part, you should still adjust for multiple comparison. That is, the code should be contrast(emm_, "pairwise", adjust = "bonferroni"). To see why this adjustment makes the most sense, consider the case where you only have 1 dependent and 1 independent variable. Then, the bonferroni-adjusted p-value would be $0.05/1=0.05$ and so you would proceed as if there were no correction. In other words if you don't adjust for multiple testing in the pairwise comparison in your case, you would never adjust for multiple testing in any pairwise comparison.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.