1
$\begingroup$

I am doing final revisions to my dissertation, which includes a large number of regressions. I have six outcome variables (DVs) and I am testing the effect of four primary (correlated) IVs, but I am testing them separately to determine the unique contribution to the model of each IV. Therefore, I have 24 regressions.

One of my committee members would like me to use Bonferroni correction to control for Type I error; however, I am struggling with determining the number to use to determine the significance threshold. I guess the main thing I'm struggling with is that while I include these 24 regressions in my dissertation, any publication that is derived from my dissertation will not include all of these analyses, so setting the Bonferroni correction to .05/24=.002 seems (1) extremely conservative and (2) would be less conservative in a subsequent journal publication, as I would be correcting for fewer tests.

Are there other ways to approach determining the number of "tests" that alleviate Type I error concerns and make theoretical sense without being so conservative? I've been struggling to find some useful resources/articles for the last few days that account for a situation like mine, but most discussions of Bonferroni (or Holm-Bonferroni or other derivatives) focus on treatment of a single regression or analysis rather than a range of analyses across a dataset. I also realize that this is one of the biggest criticisms of Bonferroni--that the idea of constantly adjusting the alpha with each test would (in theory) require us to put off publishing anything until we had exhausted a given dataset.

Any links to useful articles or advice on how to approach my current problem would be greatly appreciated!

Improve this question
$\endgroup$
2
  • $\begingroup$ I should note that my analyses are of human behavioral/perceptual data (I'm a social scientist), so the measures that I'm including in my regressions are Likert-type scales measuring perceptions of factors like relational closeness (one of my DVs) and engagement in relationship maintenance strategies (my IVs). I'm not sure if that would impact the type of procedure I perform (vs. if these data were non-perceptual, for example). $\endgroup$
    – Jessica
    Dec 20, 2012 at 16:03
  • 2
    $\begingroup$ One of the major conceptual points behind the Bonferroni correction is that you should be counting all the tests you do, whether or not you publish their results. Otherwise your research is no better than the green jelly bean scientists. $\endgroup$
    – whuber ♦
    Dec 20, 2012 at 16:42

1 Answer 1

Reset to default
1
$\begingroup$

To adjust for multiple testing, instead of controlling the family-wise error rate as Bonferroni adjustment does, you could control the false discovery rate, for example by means of the Benjamini-Hochberg procedure. While this is qualitatively different - you no longer control the proportion of false positives of all tests you did, but the proportion of false positives of the tests you did that turned out positive - it is much less conservative.

Judgement of whether this is appropriate in your situation would require more detailed knowledge of your application.

The wikipedia article might be a good place to get an overview.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the reply. I conduct two series of analyses. In the first, I run four sets of regressions, each with a different relational outcome (e.g., relational closeness). For each DV, there are four regressions, each with a set of control and behavioral IVs as well one of four relationship maintenance strategies. These strategies are correlated, so they cannot be included in the same regression, and I want to see how each individually relates to the DV. $\endgroup$
    – Jessica
    Dec 20, 2012 at 16:36
  • $\begingroup$ There is then a second set of regressions with two other DVs that includes the same IVs, plus a DV from the previous regression (relational closeness) as a control plus tests for interaction effects between the relationship maintenance strategies & relational closeness. So in sum, this leads to 24 regressions (6 DVs x 4 regressions per DV). $\endgroup$
    – Jessica
    Dec 20, 2012 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.