# What do we call multiple testing?

I can think of different “types” of multiple testing when using linear models for example:

1. Multiple inferences because we have several dependent variables
2. Multiple inferences because we have several independent variables
3. Looking at the data without making any test. Running a test on only the comparisons that might possibly yield to a significant p.value.
4. Running multiple different tests on the same data. (try a LM, if it is not significant, try a GLM, if it is still not significant try a beta regression, etc.)

Wikipedia says:

[...] multiple testing problem occurs when one considers a set of statistical inferences simultaneously or infers a subset of parameters selected based on the observed values.

Does the first part of what wikipedia says encompass my first two points and the part that I put in italics (after the "or") is equivalent to my third point ? Is it correct that my point 4 has nothing to do with what we call multiple testing ?

If my question is too blurry, I might rephrase it this way:

When does an issue of multiple testing occur ? How would you categorize (if needed) the possible events of multiple testing ?

• "Wikipedia says" is not an acceptable reference. I imagine that pre-university students are told this in many educational systems. Please give a full reference. Jul 16, 2013 at 12:16
• Do you complain about the absence of the link or about the fact that the source is not trustworthy ? I'll add the link hoping you'll be fine with it. Thks Jul 16, 2013 at 12:19
• The absence of a link is the primary issue. I wouldn't prejudge the accuracy without reading it. Jul 16, 2013 at 12:20
• Note that you can have multiple independent variables (point 2) and still conduct only one test (comparing two models, testing all predictors simultaneously).
– Gala
Jul 16, 2013 at 14:04
• I have written a little guide, in the hope of making sense of multiple testing: arxiv.org/abs/1304.4920 Jul 16, 2013 at 19:00

## 2 Answers

If all aspects of a test are specified in advance and its assumptions are met, you can safely conclude that the null hypothesis will be rejected erroneously at the frequency defined by the error level. If you conduct several tests (a “family” of tests), each of these tests is an additional occasion to commit this error.

Each individual test might still have its nominal error level but the probability that you reject at least one null hypothesis erroneously in the family will be higher. To the extent that you had reasons to set an error level in the first place, this is a problem as the probability to commit at least one error is higher than said error level. This is the core of the concern about multiple testing and it seems to apply to all four situations you describe.

Now, if the tests are independent and all null hypotheses are true, you know what the probability of committing at least one error over the whole family is (incidentally, you also know that any rejection must be erroneous). If they are not independent or some of the null hypotheses are in fact not true, not only is the actual family-wise error level higher than the nominal level but it's difficult to know exactly how high (you can however put bounds on it; that's the reasoning behind the Bonferroni adjustment). If the various hypotheses are related in some way, specific solutions might apply (for example classical “multiple comparison” techniques, multivariate tests, sequential procedures in clinical trials) but even if they do not, the problem is still there.

Repeatedly testing as you collect data (also known as optional stopping or “sampling to a foregone conclusion”), trying various techniques, analyzing various subsamples or dependent variables also expose you to multiple testing issues. These situations are not always discussed together but there is no reason why they should not. Different techniques testing the same hypothesis or related ones (your point 4) might be closely related and possibly not entail as much of an increase in the family-wise error level as multiple tests on completely unrelated samples but you are still conducting several tests.

Possibly the most delicate issue is point 3. In such a setting, you could very well run a single statistical test. How could that lead to a multiple testing problem? One argument in favor of this view is that the p-value depends on the distribution of a test statistic over hypothetical replications. If you would replicate this experiment, you would carry out a different test each time depending on how the data “look”. The distribution of this test statistic would not be the same as if you would blindly test the same comparison every time as it is also influenced by this prior informal visual inspection of the data. In fact, you are implicitly considering many possible comparisons in your study, a multiple testing situation.

A similar reasoning also applies to the situation described in point 4. It might or might not correspond to what is usually called “multiple testing issues” (the perennial problem of is-this-really-called-X questions) but the consequence is the same: The tests are uninterpretable as they could be far from their nominal error level. The situation is further muddled by the fact that you propose conducting further tests based on the results of the earlier ones but you are in any case willing to run multiple tests. (Note that this is based on the fact that you claimed to make decisions based on significance alone. Picking a model based on the residuals or some other diagnostics and only conduct one significance test seems a much better approach.)

My reasoning on the last two points is inspired in particular by Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin and Review, 14 (5), 779-804.

Items 3 and 4 on your list appear to be most closely related to the problem of controlling the error rate. I hope someone else will comment on items 1 and 2. I have never quite known how to think of multiple testing of regression coefficients from the same model.

Even if you don't actually test the data, but visually look for patterns that you can then test, you will have an error rate problem. Quantifying it may be hard, but it's there.

The essence is that by random chance you will find patterns in most data sets. The hard part is making the case that they are real (i.e., exist in the population from which the data came) rather than a chance function of a random draw from the population. It's easier to claim they are real if you haven't scoured the data for patterns.

Scientists concerned with publication (or more generally concerned with getting small $p$-values) who do not control the error rate and look for patterns around which a story can be built are said to be capitalizing on chance. They are taking advantage of the fact that patterns, random fluctuations or not, can always be found. This can happen consciously or subconsciously.