18
$\begingroup$

I am using a generalized linear model in SPSS to look at the differences in average number of caterpillars (non-normal, using Tweedie distribution) on 16 different species of plants.

I want to run multiple comparisons but I'm not sure if I should use a Sidak or Bonferroni correction test. What is the difference between the two tests? Is one better than the other?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ I hate the fact that such corrections are often needed with standard frequentist hypothesis testing and I much prefer Bayesian techniques. That said, I hate the Sidak correction less because it seems less ad-hoc (if you are willing to accept the assumption of independence). This is mostly just personal preference though so I made it a comment instead of an answer. $\endgroup$
    – Michael McGowan
    Jan 9, 2012 at 18:07
  • 1
    $\begingroup$ @MichaelMcGowan: Just curious, but, what do you consider "ad hoc" about a Bonferroni correction? $\endgroup$
    – cardinal
    Jan 9, 2012 at 19:42
  • $\begingroup$ @cardinal Sorry, that probably wasn't the best choice of words. At the cost of needing stronger assumptions (I don't want to trivialize that cost), the Sidak correction creates a bound with more qualitative meaning. I can't really qualitatively explain what the bound represents in the Bonferroni correction aside from a sort of worst-case bound according to Boole's inequality. $\endgroup$
    – Michael McGowan
    Jan 9, 2012 at 20:54
  • $\begingroup$ @MichaelMcGowan: Ah, ok. I see. I suppose there are a couple of qualitative things one could say about Bonferroni: (a) It provides guaranteed protection against the familywise error rate, regardless of the dependence between the individual test statistics under the null and (b) It is the exactly correct correction to make when the rejection regions of the individual hypothesis tests are pairwise disjoint. $\endgroup$
    – cardinal
    Jan 9, 2012 at 21:10
  • 1
    $\begingroup$ Two tests aren't independent if the probability of a type I error for one test correlates with that for the other test. For example, suppose you run an experiment with one control condition and two test conditions. The two tests comparing each test condition to the control condition are not independent. You can see this by considering what happens if you by chance get an extreme value for the control condition. This would make both of the two tests more likely to be statistically significant. $\endgroup$
    – user32473
    Nov 7, 2013 at 21:04

4 Answers 4

Reset to default
25
$\begingroup$

If you run $k$ independent statistical tests using $\alpha$ as your significance level, and the null obtains in every case, whether or not you will find 'significance' is simply a draw from a random variable. Specifically, it is taken from a binomial distribution with $p=\alpha$ and $n=k$. For example, if you plan to run 3 tests using $\alpha=.05$, and (unbeknownst to you) there is actually no difference in each case, then there is a 5% chance of finding a significant result in each test. In this way, the type I error rate is held to $\alpha$ for the tests individually, but across the set of 3 tests the long-run type I error rate will be higher. If you believe that it is meaningful to group / think of these 3 tests together, then you may want to hold the type I error rate at $\alpha$ for the set as a whole, rather than just individually. How should you go about this? There are two approaches that center on shifting from the original $\alpha$ (i.e., $\alpha_o$) to a new value (i.e., $\alpha_{\rm new}$):

Bonferroni: adjust the $\alpha$ used to assess 'significance' such that

$$\alpha_{\rm new}=\frac{\alpha_{o}}{k}\qquad\qquad\quad$$

Dunn-Sidak: adjust $\alpha$ using

$$\alpha_{\rm new}=1-(1-\alpha_{o})^{1/k}$$

(Note that the Dunn-Sidak assumes all the tests within the set are independent of each other and could yield familywise type I error inflation if that assumption does not hold.)

It is important to note that when conducting tests, there are two kinds of errors that you want to avoid, type I (i.e., saying there is a difference when there isn't one) and type II (i.e., saying there isn't a difference when there actually is). Typically, when people discuss this topic, they only discuss—and seem to only be aware of / concerned with—type I errors. In addition, people often neglect to mention that the calculated error rate will only hold if all nulls are true. It is trivially obvious that you cannot make a type I error if the null hypothesis is false, but it is important to hold that fact explicitly in mind when discussing this issue.

I bring this up because there are implications of these facts that appear to often go unconsidered. First, if $k>1$, the Dunn-Sidak approach will offer higher power (although the difference can be quite tiny with small $k$) and so should always be preferred (when applicable). Second, a 'step-down' approach should be used. That is, test the biggest effect first; if you are convinced that the null does not obtain in that case, then the maximum possible number of type I errors is $k-1$, so the next test should be adjusted accordingly, and so on. (This often makes people uncomfortable and looks like fishing, but it is not fishing, as the tests are independent, and you intended to conduct them before you ever saw the data. This is just a way of adjusting $\alpha$ optimally.)

The above holds no matter how you you value type I relative to type II errors. However, a-priori there is no reason to believe that type I errors are worse than type II (despite the fact that everyone seems to assume so). Instead, this is a decision that must be made by the researcher, and must be specific to that situation. Personally, if I am running theoretically-suggested, a-priori, orthogonal contrasts, I don't usually adjust $\alpha$.

(And to state this again, because it's important, all of the above assumes that the tests are independent. If the contrasts are not independent, such as when several treatments are each being compared to the same control, a different approach than $\alpha$ adjustment, such as Dunnett's test, should be used.)

Improve this answer
$\endgroup$
6
  • $\begingroup$ +1. Is what you call a "step-down" approach for Bonferroni exactly equivalent to what is known as Holm-Bonferroni method? If yes, then does the same logic applied to Dunn-Sidak have a name? $\endgroup$
    – amoeba
    Dec 1, 2015 at 22:35
  • 2
    $\begingroup$ @amoeba, yes it is sometimes called "Holm's method", hence Holm-Bonferroni or Holm-Sidak. $\endgroup$
    – gung - Reinstate Monica
    Dec 2, 2015 at 3:23
  • $\begingroup$ Thanks. Another question I have is about your statement that if you are running theoretically-suggested, a priori, orthogonal contrasts, you don't usually adjust $\alpha$. How important is "orthogonal" in here? E.g. if you have 6 subject groups and compare groups 2, 3, 4, 5, and 6 to group 1 (where group 1 might e.g. be a control group), then these are non-orthogonal contrasts. Would you feel different about adjusting $\alpha$ in this case than when your contrasts are indeed orthogonal, like 1-2, 3-4, 5-6? If so, why? $\endgroup$
    – amoeba
    Dec 2, 2015 at 10:24
  • $\begingroup$ @amoeba, running 3 a-priori, orthogonal contrasts in 1 study isn't any different than running 1 a-priori contrast in each of 3 different studies. Since no one argues that you need familywise corrections for the latter, there is no coherent reason to require them for the former. In your other example, if the control group should bounce lower by chance alone, every one of your 5 contrasts will look good; but that is unlikely to happen if you ran 5 independent studies. You should really use some form of adjustment, or you could use Dunnett's test. $\endgroup$
    – gung - Reinstate Monica
    Dec 3, 2015 at 2:43
  • $\begingroup$ I don't think I fully understand. I ran a quick simulation with $\mathcal N(0,1)$ values in each group with $n=10$ and $\alpha=0.05$. I get 0.14 chance of at least one false positive for three orthogonal contrasts and 0.12 chance for three non-orthogonal contrasts as above. That's very close. The difference is much larger for the chance of getting all three false positives: 0.0001 and 0.002. So I understand that getting several significant outcomes is much more likely with non-orth. contrasts, but if one is concerned with familywise error rate, then the two cases seem to be almost identical. $\endgroup$
    – amoeba
    Dec 3, 2015 at 11:10
9
$\begingroup$

Denote with $\alpha^*$ the corrected significance level, then Bonferroni works like this: Divide the significance level $\alpha$ by the number $n$ of tests, i.e. $\alpha^*=\alpha/n$. Sidak works like this (if the test are independent): $\alpha^*=1 − (1 − \alpha)^{1/n}$.

Because $\alpha/n < 1 − (1 − \alpha)^{1/n}$, the Sidak correction is a bit more powerful (i.e. you get significant results more easily) but Bonferroni is a bit simpler to handle.

If you need an even more powerful procedure you might want to use the Bonferroni-Holm procedure.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Why is Bonferroni simpler to handle? $\endgroup$
    – Emily
    Jan 9, 2012 at 17:36
  • 4
    $\begingroup$ I find dividing $\alpha$ by $n$ algebraically simpler than calculating $1-(1-\alpha)^{1/n}$, but I am lazy. Also Bonferroni does not assume indenpence hence it is "simpler" in the sense of assuming less. But you pay the price of it being more conservative. $\endgroup$
    – Momo
    Jan 9, 2012 at 17:56
  • $\begingroup$ @Momo Computers are really, really good at arithmetic, so I don't find the simplicity argument very compelling. A hundred years ago when calculations were being done by hand was a very different story of course. $\endgroup$
    – Michael McGowan
    Jan 9, 2012 at 18:10
  • $\begingroup$ +1 compared to my answer, this gets to the point quite succinctly ;-). $\endgroup$
    – gung - Reinstate Monica
    Jan 10, 2012 at 4:23
  • $\begingroup$ Haha that's what I thought you meant! Thanks so much! $\endgroup$
    – Emily
    Jan 10, 2012 at 15:52
6
$\begingroup$

Sidak and Bonferroni are so similar that you will probably get the same result regardless of which procedure you use. Bonferroni is only marginally more conservative than Sidak. For instance, for 2 comparisons and a familywise alpha of .05, Sidak would conduct each test at .0253 and Bonferroni would conduct each test at .0250.

Many commenters on this site have said that Sidak is only valid when the test statistics of your comparisons are independent. That's not true. Sidak allows slight inflation of the familywise error rate when the test statistics are NEGATIVELY dependent, but if you're doing two-sided tests, negative dependence isn't generally a concern. Under non-negative dependence, Sidak does in fact provide an upper bound on the familywise error rate. That said, there are other procedures that provide such a bound and tend to retain more statistical power than Sidak. So Sidak probably isn't the best choice.

One thing the Bonferroni procedure provides (that Sidak doesn't) is strict control of the expected number of Type I errors--the so-called "per-family error rate," which is more conservative than the familywise error rate. For more info, see: Frane, AV (2015) "Are per-family Type I error rates relevant in social and behavioral science?" Journal of Modern Applied Statistical Methods 14(1), 12-23.

Improve this answer
$\endgroup$
5
$\begingroup$

The Sidak correction assumes the individual tests are statistically independent. The Bonferroni correction doesn't assume this.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Does that mean that the Bonferroni is simply a more conservative test? $\endgroup$
    – Emily
    Jan 9, 2012 at 17:37
  • 1
    $\begingroup$ Bonferroni is more conservative when both tests are appropriate. But if your tests aren't independent, you shouldn't use Sidak. $\endgroup$
    – onestop
    Jan 9, 2012 at 18:26
  • 2
    $\begingroup$ +1 That the Bonferroni correction doesn't require the tests to be independent is a good point that I didn't cover. $\endgroup$
    – gung - Reinstate Monica
    Jan 10, 2012 at 4:21
  • $\begingroup$ @onestop: What does it mean that the tests are independent? Could you perhaps give an example? $\endgroup$
    – Gunnhild
    Jul 15, 2013 at 14:10
  • 2
    $\begingroup$ The Sidak correction doesn't require independence. It only assumes the tests are not negatively dependent. Positive dependence is fine. $\endgroup$
    – Bonferroni
    Aug 3, 2016 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.