5
$\begingroup$

I have run 6 independent-samples t-tests, and 4 Mann Whitney U tests.

When applying Bonferroni correction, does it apply across all 10 tests?

My hypotheses are that there will be 4 significant differences for the independent-samples t-tests, and 3 for the Mann Whitney U tests.

The sample sizes are unequal, with 19 participants and 63 participants respectively. Mann Whitney U tests were performed on Likert-style mood data. Independent t-tests were performed on a self-report scale, and its various subscales.

$\endgroup$
2
  • $\begingroup$ Suppose the six tests are labelled A, B, ... F. Have you predicted that (for example) A, B, D, and F are the ones which are significant or just any 4 from 6? $\endgroup$
    – mdewey
    Commented May 2, 2016 at 16:28
  • $\begingroup$ Yes, but are you specifying exactly which four or just any four? $\endgroup$
    – mdewey
    Commented May 3, 2016 at 8:36

3 Answers 3

1
$\begingroup$

As umm... Mr. Bonferroni notes, his correction works on any p-values, regardless of their source. However, there are other procedures, like the Holm's, which are uniformly more powerful; subject to certain other restrictions, like positive dependence between the tests, other methods are even more powerful still. Sorry, Carlo.

These corrections are intended to preserve the familywise error rate, which essentially means that you want to keep the probability of making one or more errors within a "family" of tests at the same level you would accept for a single test. To do this intelligently, you need to define the families appropriately. Based on your description, it sounds like you have at least two families of tests: the first consisting of mood data and the 2nd consisting of the self-report scales. For example, you might be testing whether some manipulation causes subjects in the treatment group (vs. appropriate controls) to 1) experience a change in mood and 2) be aware of said change.

Accordingly, I'd consider adjusting the first set with $n=6$ and the latter with $n=4$. Since the one of the self-report values is the overall scale, you could potentially argue that it "protects" the three sub-scores too, so perhaps I'd consider reporting the overall score (uncorrected) and three sub-scores corrected with $n=3$, especially if the overall test is significant. If the Likert (or self-report) scales all attempt to measure the same thing, I'd be tempted to apply an omnibus test (e.g., an ANOVA) first, which might give you more power.

However, I think applying the multiple comparisons to both sets of data (with $n=10$ is over-conservative) unless these tests are intimately linked.

$\endgroup$
2
  • 1
    $\begingroup$ In fact he did not invent the method so no need to apologise to him. It is based on Boole's Inequality and was introduced into the field of simultaneous statistical inference in a series of papers by Dunn. $\endgroup$
    – mdewey
    Commented May 3, 2016 at 8:39
  • $\begingroup$ @mdewey, Stigler's law strikes again! Still, it feels rude not to apologize since he's posting in this thread :-) $\endgroup$ Commented May 3, 2016 at 16:35
5
$\begingroup$

Bonferroni works for any p-values. It doesn't care where they came from.

$\endgroup$
1
  • 3
    $\begingroup$ Well, this is a great novelty account :) $\endgroup$
    – Neil G
    Commented May 1, 2016 at 16:31
1
$\begingroup$

If you want to correct then there are stronger methods than that named after Bonferroni like Holm's method which is always applicable when Bonferroni is and is never less powerful. However is that really what you want to do if you have a precise hypothesis about which will achieve some arbitrary level of statistical significance?

$\endgroup$
2
  • $\begingroup$ Perhaps that's part of the problem. I'm not really sure whether I should be applying post hoc or not. I was under the assumption that multiple tests = apply bonferroni. When is it actually applicable? $\endgroup$ Commented May 1, 2016 at 17:38
  • $\begingroup$ Suppose you have done 1000 tests of the same hypothesis. In order to protect you from the consequences of finding many results by chance you would use some form of adjustment. If you edit your question to give more detail I will try to edit my answer to help. $\endgroup$
    – mdewey
    Commented May 2, 2016 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.