Do I have a multiple comparisons scenario here?

Question

Edited after the first 2 responses, additional content in italics

I have 4 treatments and a control to encourage people to sign-up for a class. I'll be looking at sign-up and actual attendance rates, with covariates of age, nationality, and gender.

There are 3 main things I want to do:

1. I want to find out whether any of these treatments work (i.e., do they trigger higher signup rates?, compared to a simple control?)
2. I want to evaluate whether there are other variables that affect efficacy of these treatments (i.e., run multiple regressions to control for confounders)
3. I want to find out how treatment effectiveness varies by subgroups e.g. for older people, what treatment works the best? For people of certain nationalities, what treatment works the best?

Problem I can't wrap my head around the concept of multiple comparisons. Since I'm doing 3 sets of analyses, there are technically plenty of hypotheses tests/comparisons going on (e.g. testing whether treatment coefficients in varying contexts are statistically different from 0).

Q1: Do I have a multiple comparisons problem, such that I need to do a Bonferroni/Sidak/FDR correction?

Q2: Are my tests are all part of a 'family'? (and what even defines a family?)

Appreciate your guidance.

Answer 1

As a preliminary I would say that direct answers to your questions depend on several things that are not clear from your question, and that you have quite likely not considered. Also, discussions about multiple testing often include some seemingly incompatible opinions. For example, a strict "frequentist" might say that they are necessary but a Bayesian that they are unnecessary.

"Correction" for multiple comparisons is only needed if you want to control the global false positive error rate within the Neyman–Pearsonian framework that fixes false positive error rates in the long run. The long run error performance relates to the test procedure rather than the data and so it can be thought of as 'global'.

An alternative approach, sometimes called 'neo-Fisherian' (or 'neoFisherian' https://www.jstor.org/stable/23736900) focusses on the local evidence in the dataset regarding the particular null hypothesis of interest. The data contain the evidence and as the data come from this study (i.e. not from any notional 'long run') we can think of an evidential evaluation as 'local' in contrast to the 'global' long run error rates.

Both the Neyman–Pearsonian and neo-Fisherian approaches are fully 'frequentist' because they are both derived from the interpretation of probability as a long run frequency rather than something like a fractional level of belief or certainty. The contrast to the frequentist methods are Bayesian. You do not seem to be asking about Bayesian methods and so I will not attempt to describe them here beyond saying that they are like the neo-Fisherian in being concerned with local evidence rather than global long run errors. (I am not experienced with Bayesian approaches.)

The best approach to use depends on the nature of inferences that you wish to make and, critically, on whether the study has to stand independently as definitive. If the latter is the case then consider using a Neyman–Pearsonian decision procedure and deal with your question of what constitutes a family after delineating carefully the most critical inferences. If your study is in any sense preliminary or hypothesis generating then the neo-Fisherian approach is advantageous because it allows you to rank the interventions and to quantitate the relevant evidence. That will allow sensible design of follow-up studies (to be done by you or others). In my opinion, most studies and most parts of a study can be thought of as preliminary and so the neo-Fisherian approaches are most useful.

I know that this answer may not directly help you, but it might help you to get closer to "wrap [your] head around" the real issues. I have an extensive chapter that you will find relevant here: https://link.springer.com/chapter/10.1007/164_2019_286 . There is a section that directly addresses multiple comparisons, but you probably should look at the whole thing.

Answer 2

I agree with @MichaelLew that you have not given enough details for a definitive answer. However, let's explore one possible specific scenario. It may answer your question and it may not. If not please say what is wrong or incomplete about my scenario, and maybe someone can use that information to provide what you need.

---

Suppose you have 200 subjects in each of the five groups. where '5' is the control group. In each group, some sign up for the class (Yes) and some don't (No), giving a table similar to the fictitious one below.

Group     1     2     3    4    5    Total
------------------------------------------
Yes      78    55    98   67   50      348
No      122   145   102  133  150      652
------------------------------------------
Tot     200   200   200  200  200     1000

Yes = c(78,55,98,67,50)
sum(Yes)
[1] 348
Tot = rep(200, 5)
No = Tot-Yes;  No
[1] 122 145 102 133 150

TAB = rbind(Yes, No);  TAB
    [,1] [,2] [,3] [,4] [,5]
Yes   78   55   98   67   50
No   122  145  102  133  150

A chi-squared test of homogeneity tests the null hypothesis that, in the population from which these 1000 subjects were chosen, all five groups have the same probability of giving Yes responses.

This is done by comparing the observed counts $X_{ij}$ in TAB with expected counts computed, according to the null hypothesis, from row and column totals. For example expected count $E_{11} = (348)(127)/1000 = 44.196.$

In R, the chi-squared test (without continuity correction due to relatively large counts), is as shown below. The P-value very near $0$ indicates strong evidence that the groups have significantly different proportions of Yes's.

chisq.test(TAB, cor=F)

        Pearson's Chi-squared test

data:  TAB
X-squared = 32.641, df = 4, p-value = 1.415e-06

Now the question arises, which groups differ from which others. The Pearson residuals having the largest absolute values. )The chi-squared statistic is the sum of the squares of the ten residuals.) Here, it seems especially worthwhile comparing Groups 3 and 5.

chisq.test(TAB, cor=F)$res
          [,1]      [,2]      [,3]       [,4]      [,5]
Yes  1.0068729 -1.750041  3.404189 -0.3116511 -2.349370
No  -0.7355979  1.278539 -2.487022  0.2276851  1.716395

This ad hoc comparison can be made using chisq.test again, but with a sub-table of TAB, limited to columns 3 and 5,

TAB[, c(3,5)]
    [,1] [,2]
Yes   98   50
No   102  150

chisq.test(TAB[ ,c(3,5)], cor=F)

        Pearson's Chi-squared test

data:  TAB[, c(3, 5)]
X-squared = 24.71, df = 1, p-value = 6.662e-07

For your actual data, you can decide how many such ad hoc comparisons to make. However, if you are going to make five such comparisons you should use a smaller significance level than 5% to avoid 'false discovery, making multiple tests on the same data.

The Bonferroni method of avoiding false discovery would divide 5% by 5 comparisons, so you should find any one comparison to be significant only if the P-value of the ad hoc test is smaller than 1%.