I understand when performing a simple t-test, we typically control the type -1 error rate at $\alpha = .05$. This signifies that if the null hypothesis holds, the data will "incorrectly" reject the null hypothesis in 5% of all instances. Therefore, if we are performing 100 consecutive tests (and all of which the null hypothesis holds), we will reject the null in 5 of these tests. Thus, the need for multiple correction adjustment.

Currently, my statistics professor disagrees with my analysis. I will provide an example and then extrapolate further out:

  1. My data frame contains two groupings of people (say, type A and type B). For each person we have 100 different measurements (things ranging from: height, weight, IQ, etc...). I want to run 100 different t-tests comparing the means of each of my covariates among Type A and type B people. Am I still required to use a multiple correction scheme here? I say no because these are different measurements between groups A and B. They say yes because I am performing multiple tests.

  2. I happen to be the only statistician in my town. I am contracted to perform 100 different analyses (for 100 different researchers). Each researcher asks me to complete a single t-test. Do I need to perform a multiple test correction here? I would argue no though I fail to see the difference between this example and the previous.

  • $\begingroup$ An important difference is that in (1) there is one sample of subjects, while in (2) there is a different sample of subjects for each of the researchers. $\endgroup$
    – BruceET
    Oct 27 '19 at 18:26
  • 2
    $\begingroup$ This older thread was more focused on reproducibility, but might also be relevant: stats.stackexchange.com/questions/206592/… $\endgroup$ Oct 27 '19 at 19:05

Multiple comparisons corrections are intended to control the familywise error rate--or something like it--so they should be applied across a "family" of related hypothesis tests.

In your first example, the overarching goal probably to determine whether Groups A and B differ. If you didn't control for multiple comparisons, you could trivially find an effect by adding more and more tests: if there's no difference in height or weight, throw in BMI, Zodiac Sign, and annual ice cream consumption. Eventually, one of these tests will, by chance, cross your $\alpha$ threshold, and you'll be tempted to write something goofy like "Although As and Bs may seem similar, we observed a statistically significant difference (p=0.04) in the number of pets owned raised to the mother's age power".

In your second example, the tests are unrelated. Client #1 wants to know whether customers prefer red vs. orange trim on their cars. Client #2 wants to know if drug X shrinks tumors more than drug Y, Client #3 wants to know which factors affect her crop yields, and so on. These are all unrelated: any compound hypothesis relating them would be very bizarre.

So...how do you define "a family" of tests? I'm not sure that any hard-and-fast rules exist. Here are some guidelines:

  • Will multiple (formal) tests lead to the same substantial conclusion[*]? For example, suppose you're interested in the effects of diet on fitness. Differences in resting heart rate OR 40 yard dash time OR BMI OR bench press performance might lead you to the same conclusion: "Diet D improves fitness." If so, these tests all belong to the same family, and you ought to apply a multiple comparisons correction. T

  • Reusing the same subjects or the same hypothesis also suggests they belong to the same family. For example, maybe you test the distribution of salaries paid to men vs. women in 20 different countries. Each of those tests ought to be corrected.

[*] This requires a little subject matter expertise. For example, suppose you did 10 fitness tests, five of which could be said to measure "strength" and five of which measure "endurance." Do you apply a correction to all of the tests together, or separately within the "strength" and "endurance" families? Either might be okay, though you ought to be very clear about what you actually did. A better approach might be to generate composite scores, and sidestep the need for multiple comparisons corrections altogether.

  • $\begingroup$ I think that's precisely the answer I was looking for; I am having a hard time identifying what a "family" of tests are. Though, this does make me wonder in the second example if we would still find 5 significant results even if the null hypothesis were true in each example and how we would change that? $\endgroup$
    – Tommyixi
    Oct 27 '19 at 19:03
  • 1
    $\begingroup$ In the "consultant" example, the key is that no one cares whether "orange trim sells more cars OR drug X shrinks tumors better OR adding lime increases crop yields" is true, so there's no point in worrying about its error rate. In contrast, "drug X changes levels of protein A OR protein B or protein C" is a reasonable hypothesis, especially if those three proteins are all related somehow. $\endgroup$ Oct 27 '19 at 19:18
  • 2
    $\begingroup$ This answer hit the nail on the head with respect to controlling the family-wise error rate. However, multiple testing corrections are not always intended to control this error rate, as the equally (if not more) popular false discovery rate is something different. $\endgroup$ Oct 28 '19 at 4:12
  • $\begingroup$ I interpreted OP's example 2 slightly differently: I thought it meant that the different researchers are asking for t-tests to be performed on the same sample of people (the townspeople). So the only difference between 1 and 2 would be that there are more researchers involved, but the individual t-tests are the same. $\endgroup$
    – qdread
    Oct 28 '19 at 12:53
  • 1
    $\begingroup$ Link? I have literally never seen someone do an FDR correction across an entire work. I have seen FDR applied to a list of candidate genes or brain areas that might change due to a disease. I'd be very surprised to see a subsequent test, like whether the called gene's expression levels correlates with disease severity, included in that correction. To me, that doesn't even make sense: 5% FDR in this figure, I can understand; I can't see when I'd want--or be able to use--a guaranteed FDR over unrelated tests. $\endgroup$ Oct 29 '19 at 0:55

The only real answer is - whenever you want to, as long as you can justify it. A reasonable case can be made for not adjusting at all. A reasonable case can be made for doing different things in different cases. This is more generally true of statistical analysis than most people think (see Statistics as Principled Argument by Robert Abelson for a lot more

First, recognize that 5% is arbitrary. Why not 1%? Why not 10%? Why is the usual power set at 80% or 90% instead of 95% or 99%?

Second, taking a tiny bit of a Bayesian view, how sure are we that the null is false? Except for some categorical tests with small populations, the null is never exactly true. If you (say) compare the heights of people with even and odd social security numbers and somehow get data for the entire population of the USA, the means will be different. Maybe in the 4th or 5th decimal place, but they won't be identical. But, in most research, we are pretty sure that the null isn't even approximately true.

The whole structure is an unfortunate legacy of Fisher's situation: He was testing different treatments of plots of land at Rothamstead. In this case, the notion that the null might be approximately true was sensible. Is treatment A better than treatment B? He had no real idea - so, he tested. But, often, we do know. If we are testing a medical treatment, for instance, we can be pretty sure it has some effect. The more certain we are, a priori that the null is not close to true, the less inclined we should be to adjust p values. Also, if recruiting more subjects and doing the measurements is expensive, we may think twice about adjusting. But if we have "big data" then it's no problem.

Third, remember that decreasing type I error will either increase type II error or require a bigger sample. Is a type I error worse than a type II error? The usual choices of 5% and 20% indicate that we think so. But sometimes type II error is much much worse. Suppose we have a drug that we think might reverse some disease which is terminal and currently untreatable. Then a type I error means giving an ineffective drug to a dying person while a type II error means letting someone die who could be cured.

Finally, remember that the whole significance testing apparatus is increasingly criticized, with varying degrees of severity. I'm a pretty harsh critic (you may have figured that out by now!) but who am I? Just some guy on Cross Validated. But the American Statistical Association has (to a much lesser degree) joined the critics.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.