6
$\begingroup$

In our randomized controlled trial, we used linear mixed effects models to test differences between groups in changes from baseline to six months while adjusting for important covariates. We ran separate analyses for each outcome. We had 6 covariates that were considered, and we used a stepwise method using Akaike information criterion for selection of the best variable set.

A reviewer came back to us saying that we should have addressed multiplicity and adjusted our p-values due to a possibility of inflating type I error.

I am not sure if this is true or not, but I don't think that we should do this, because we did not conduct post hoc analysis. Also, in our analysis we did not have multiple levels of treatment. So, I don't really think we should adjust our p-values. The only thing that we should adjust for, I believe, is for selecting the covariates. But, that was taken care of by using iterative model selection technique, namely stepwise variable selection based on AIC (stepAIC).

P.S. I searched the site for possible answers, and couldn't find any that fits.

$\endgroup$
  • $\begingroup$ What is your outcome and what are the 6 covariates ? Did you use the default k=2 option in stepAIC ? $\endgroup$ – Robert Long Jul 27 '16 at 20:03
  • $\begingroup$ @RobertLong We had two continuous outcomes, and 6 covariates: age, weight, height, classification (4 levels), QScore (continuous variable) and BYN (dichotomous: yes/no). I think, we used the default option in stepAIC. $\endgroup$ – Sanja Jul 27 '16 at 20:35
  • $\begingroup$ 2 outcomes ? So you had 2 separate models ? Were those covariates unbalanced across the groups / potential confounders, or strongly predictive of the outcome ? Why were they "important" covariates in the first place ? Did you adjust for baseline ? $\endgroup$ – Robert Long Jul 27 '16 at 22:29
  • 1
    $\begingroup$ In that case I don't think you should discard any variables. Removing variables based on some p-value criterion is a very bad idea. $\endgroup$ – Robert Long Jul 29 '16 at 15:17
  • 1
    $\begingroup$ Yes, exactly, you should retain them all, unless there is a good reason to remove any of them (such as severe multicollinearity, or if any are mediators). A non-significant p-value is not a good reason. $\endgroup$ – Robert Long Jul 29 '16 at 17:19
3
$\begingroup$

Since the reviewer only seems to be concerned about the two outcomes measured on the same subjects (and did not question the modeling procedure itself), I would simply use a sequential Bonferroni adjustment (a.k.a. Holm-Bonferroni method) to correct for it.

  1. Sort your $p$-values in ascending order
  2. Refer to them as $p_i$ (i.e. $p_1, p_2, p_3$, etc.)
  3. Than you adjust your $\alpha$-level and compare the $p$-values against that new $\alpha$-levels, i.e. you test whether $p_i \le \alpha / (1 + k - i)$, where $k$ is the number of statistical tests conducted, i.e. the number of $p$-values calculated. You can stop when $p_i \gt \alpha / (1 + k - i)$. Those $p_i$ that fall below the sequentially adjusted $\alpha$-levels are now your significant tests which are adjusted for multiplicity (after the Holm-Bonferroni method).

For example you conducted five tests ($\alpha = 0.05$) resulting in the following $p$-values:

$p_1 = 0.0024, p_2 = 0.0084, p_3 = 0.019, p_4 = 0.027, p_5 = 0.12$

The new $\alpha$-level you compare $p_1$ against is:

$0.05/(1+5-1) = 0.01$

Since $p_1 \le 0.01$ you can move on to $p_2$:

$0.05/(1+5-2) = 0.0125$

Since $p_2 \le 0.0125$ you can move on to $p_3$:

$0.05/(1+5-3) = 0.0167$

Since $p_3 \gt 0.0167$ you can stop.

In this case, from initially four significant $p$-values, you now only have two but those are adjusted for multiplicity (Note: Instead of adjusting the $\alpha$-levels, you can also adjust the $p$-values and compare against your chosen $\alpha$-level (e.g. $\alpha = 0.05$). Then all you need to do is $(1 + k - i)*p_i$ instead).

See also:

Abdi, H. (2010). Holm’s sequential Bonferroni procedure. Encyclopedia of research design, 1.

Peres-Neto, P. R. (1999). How many statistical tests are too many? The problem of conducting multiple ecological inferences revisited. Marine Ecology Progress Series, 176, 303-306.

Alternatively, you could also argue that you don't want to adjust for multiplicity because of reason such as being concerned with making type II errors.

See here: Feise, R. J. (2002). Do multiple outcome measures require p-value adjustment?. BMC Medical Research Methodology, 2(1), 1.

Or maybe this one: Gelman, A., Hill, J., & Yajima, M. (2012). Why we (usually) don't have to worry about multiple comparisons. Journal of Research on Educational Effectiveness, 5(2), 189-211.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.