1
$\begingroup$

I have 42 DV's that I have individually modelled versus the same single IV - using GAMs via the mgcv package in R. This has produced a total of 42 individual models for each of my DV's and the associated p-vlaues.

My question is: should I be adjusting p-values under the above circumstances? I understand that if one has more than 1 IV then the p-value is simply divided by the number of IV's (i.e. 0.05/20 - in the case of 20 IV's). However, I'm struggling to find any information to support this approach when there is only a single IV being considered.

$\endgroup$
2
$\begingroup$

p-values should be adjusted based on the number of independent tests, if you want to correct for multiple testing. You are referring ot the Bonferroni correction procedure, which controls the family-wise error rate (i.e. you give an upper bound for the chance that one or more null hypothesis will be rejected among $k$ true null hypotheses).

In this sense the two situations you describe are parallel and if you want to control for multile comparisons using the Bonferroni procedure you would have to proceed in thedescribed way. There are some alternatives to Bonferroni correction, such as the Šidák procedure, and some approaches that would be less conservative.

$\endgroup$
4
  • $\begingroup$ Just to clarify - this would mean calculating 0.05/42 so the p-value = 0.0012 in my case? Should this be also used even if the models are conditioned on binary (i.e. presence/absence) data? $\endgroup$ – brober Sep 21 '13 at 1:54
  • $\begingroup$ to be exact: the level of significance $\alpha$ needs to be adjusted (the p-value is calculated based on data and compared with $\alpha$. The nature of the hypothesis test does not change this rule. Be careful though, yu shuld consider the loss of power. It might be wise to decide against doing all 42 tests (in advance) unless you have enough data to do the tests with reasonable power). $\endgroup$ – jank Sep 21 '13 at 2:24
  • $\begingroup$ I have 1400 data points for each model. Could this be considered enough data not to adjust $α$? and should such an adjustment be made for models conditioned on binary data? $\endgroup$ – brober Sep 21 '13 at 3:20
  • $\begingroup$ Actually, the more data you have the more power you have for your tests. This would actually be a reason in favor of doing the adjustment, as your loss in power is offset by the sample size. Still, it also depends on your expected effect size. What exactly do you mean with "models conditioned on binary data"? $\endgroup$ – jank Sep 21 '13 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.