Why is multiple comparison a problem?

Question

I find it hard to understand what really is the issue with multiple comparisons. With a simple analogy, it is said that a person who will make many decisions will make many mistakes. So very conservative precaution is applied, like Bonferroni correction, so as to make the probability that, this person will make any mistake at all, as low as possible.

But why do we care about whether the person has made any mistake at all among all decisions he/she has made, rather than the percentage of the wrong decisions?

Let me try to explain what confuses me with another analogy. Suppose there are two judges, one is 60 years old, and the other is 20 years old. Then Bonferroni correction tells the one which is 20 years old to be as conservative as possible, in deciding for execution, because he will work for many more years as a judge, will make many more decisions, so he has to be careful. But the one at 60 years old will possibly retire soon, will make fewer decisions, so he can be more careless compared to the other. But actually, both judges should be equally careful or conservative, regardless of the total number of decisions they will make. I think this analogy more or less translates to the real problems where Bonferroni correction is applied, which I find counterintuitive.

Answer 1

You've stated something that is a classic counter argument to Bonferroni corrections. Shouldn't I adjust my alpha criterion based on every test I will ever make? This kind of ad absurdum implication is why some people do not believe in Bonferroni style corrections at all. Sometimes the kind of data one deals with in their career is such that this is not an issue. For judges who make one, or very few decisions on each new piece of evidence this is a very valid argument. But what about the judge with 20 defendants and who is basing their judgment on a single large set of data (e.g. war tribunals)?

You're ignoring the kicks at the can part of the argument. Generally scientists are looking for something — a p-value less than alpha. Every attempt to find one is another kick at the can. One will eventually find one if one takes enough shots at it. Therefore, they should be penalized for doing that.

The way you harmonize these two arguments is to realize they are both true. The simplest solution is to consider testing of differences within a single dataset as a kicks at the can kind of problem but that expanding the scope of correction outside that would be a slippery slope.

This is a genuinely difficult problem in a number of fields, notably FMRI where there are thousands of data points being compared and there are bound to be some come up as significant by chance. Given that the field has been historically very exploratory one has to do something to correct for the fact that hundreds of areas of the brain will look significant purely by chance. Therefore, many methods of adjustment of criterion have been developed in that field.

On the other hand, in some fields one might at most be looking at 3 to 5 levels of a variable and always just test every combination if a significant ANOVA occurs. This is known to have some problems (type 1 errors) but it's not particularly terrible.

It depends on your point of view. The FMRI researcher recognizes a real need for a criterion shift. The person looking at a small ANOVA may feel that there's clearly something there from the test. The proper conservative point of view on the multiple comparisons is to always do something about them but only based on a single dataset. Any new data resets the criterion... unless you're a Bayesian...

Answer 2

Well-respected statisticians have taken a wide variety of positions on multiple comparisons. It's a subtle subject. If someone thinks it's simple, I'd wonder how much they've thought about it.

Here's an interesting Bayesian perspective on multiple testing from Andrew Gelman: Why we don't (usually) worry about multiple comparisons.

Answer 3

Related to the comment earlier, what the fMRI researcher should remember is that clinically-important outcomes are what matter, not the density shift of a single pixel on a fMRI of the brain. If it doesn't result in a clinical improvement/detriment, it doesn't matter. That is one way of reducing the concern about multiple comparisons.

See also:

1. Bauer, P. (1991). Multiple testing in clinical trials. Stat Med, 10(6), 871-89; discussion 889-90.
2. Proschan, M. A. & Waclawiw, M. A. (2000). Practical guidelines for multiplicity adjustment in clinical trials. Control Clin Trials, 21(6), 527-39.
3. Rothman, K. J. (1990). No adjustments are needed for multiple comparisons. Epidemiology (Cambridge, Mass.), 1(1), 43-6.
4. Perneger, T. V. (1998). What's wrong with bonferroni adjustments. BMJ (Clinical Research Ed.), 316(7139), 1236-8.

Answer 4

To fix ideas: I will take the case when you obverse, $n$ independent random variables $(X_i)_{i=1,\dots,n}$ such that for $i=1,\dots,n$ $X_i$ is drawn from $\mathcal{N}(\theta_i,1)$. I assume that you want to know which one have non zero mean, formally you want to test:

$H_{0i} : \theta_i=0$ Vs $H_{1i} : \theta_i\neq 0$

Definition of a threshold: You have $n$ decisions to make and you may have different aim. For a given test $i$ you are certainly going to choose a threshold $\tau_i$ and decide not to accept $H_{0i}$ if $|X_i|>\tau_i$.

Different options: You have to choose the thresholds $\tau_i$ and for that you have two options:

1. choose the same threshold for everyone

2. to choose a different threshold for everyone (most often a datawise threshold, see below).

Different aims: These options can be driven for different aims such as

• Controling the probability to reject wrongly $H_{0i}$ for one or more than one $i$.

• Controlling the expectation of the false alarm ratio (or False Discovery Rate)

  What ever is your aim at the end, it is a good idea to use a datawise threshold.

My answer to your question: your intuition is related to the main heuristic for choosing a datawise threshold. It is the following (at the origin of Holm's procedure which is more powerfull than Bonferoni):

Imagine you have already taken a decision for the $p$ lowest $|X_{i}|$ and the decision is to accept $H_{0i}$ for all of them. Then you only have to make $n-p$ comparisons and you haven't taken any risk to reject $H_{0i}$ wrongly ! Since you haven't used your budget, you may take a little more risk for the remaining test and choose a larger threshold.

In the case of your judges: I assume (and I guess you should do the same) that both judge have the same budgets of false accusation for their life. The 60 years old judge may be less conservative if, in the past, he did not accuse anyone ! But if he already made a lot of accusation he will be more conservative and maybe even more than the youndest judge.

Answer 5

An illustrating (and funny) article (http://www.jsur.org/ar/jsur_ben102010.pdf) about the need to correct for multiple testing in some practical study evolving many variables e.g. functional MRI (fMRI). This short citation contains most of the message:

"[...] we completed an fMRI scanning session with a post-mortem Atlantic Salmon as the subject. The salmon was shown the same social perspective-taking task that was later administered to a group of human subjects."

that is, in my experience, a terrific argument to encourage users to use multiple testing corrections.