1
$\begingroup$

$FWER = 1-\big(1-\alpha\big)^m$

The above formula is typically used when calculating the $FWER$ where alpha is the criterion applied by the researcher to each test and $m$ is the number of tests.

However, in multiple places, I see that this only applies when the tests are statistically independent (i.e. comparing means of measurements, e.g., from groups of different/unrelated participants).

Doesn't this go against the idea that a family is: "all those experimental observations that could be analysed statistically by a global procedure’ (such as an omnibus test)" (Ludbrook, 1998). Such procedures test global null hypotheses (i.e., that all local null hypotheses are true), assuming that the observations are from the same population (e.g., having the same mean). In this sense, the individual test cannot by definition be statistically-independent because to some degree they are all evaluating the same null hypothesis.

Why would we then assume that tests are independent?

This makes no sense to me. It seems contradictory.

Ludbrook J. Multiple comparison procedures updated. Clin Exp Pharmacol Physiol. 1998;25:1032–1037.

$\endgroup$
2
  • $\begingroup$ It’s more of a bound than an equality. Think of the equation you gave as the worst-case scenario. $\endgroup$
    – Dave
    May 25, 2020 at 3:32
  • $\begingroup$ Is there any reason why we would make such worst-case scenario assumption (i.e., of fully independent pair-wise comparisons)? I am asking because in other cases in statistics when we make assumptions we make best-case scenario assumptions (e.g., when computing ANOVA we assume normally-distributed DV values within-conditions with equal variances between conditions). I guess now my question is why we sometimes assumes the worst, while other time - the best? $\endgroup$ May 25, 2020 at 3:44

1 Answer 1

2
$\begingroup$

That formula (often attributed to Šidak) is exact if the tests are independent, and it is conservative if they are positively correlated. Even when it's unreasonable to assume tests are independent, it may be reasonable to assume they are positively correlated

However, people do often use the Bonferroni bound $\textrm{FWER}= 1-m\alpha$, precisely because it makes no assumptions. The argument against the Bonferroni bound is that it's conservative. It is, but it's not very conservative for large $m$ and small $\alpha$ unless the correlation is truly extreme.

One of the first people to write about this phenomenon was Olive Jean Dunn, who said

I spent a long time trying to prove that the confidence intervals which would be used in the case of independent variables could also be used or dependent variables. After failing to find a general proof for this, I finally noticed that the simple Bonferroni intervals were nearly as short".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.