How to correctly do a nested test (t-test, rank-sum, etc)

Let's say there are two groups of participants. Each group is randomly split into control subgroup A and task subgroup B. For each of the 4 cases some random variable $$x$$ is measured, whose mean is denoted by $$\mu$$. I would like to perform 3 tests:

1. $$H_0 : \mu_{A, Group1} = \mu_{B, Group1}$$
2. $$H_0 : \mu_{A, Group2} = \mu_{B, Group2}$$
3. $$H_0 : \mu_{Group1} = \mu_{Group2}$$ regardless of subgroups

I sketch the tests below

Question: What is the correct way to perform these tests? If I perform 3 independent t-tests, is it fair to apply Bonferroni correction to their p-values, or must I do a more sophisticated multiple comparisons correction. If making independent tests is not a robust procedure, what is? I want to do all 3 tests, so as far as I understand I don't want two-way ANOVA.

• The first problem here is that your sketch does not correspond to your listed hypotheses. You sketch labels two of the tests with the same symbol ($*$), whereas your post uses numbers. Also, the null hypotheses in your post look like they should be the alternative hypotheses.
– Ben
Commented Sep 10, 2021 at 15:24
• @Ben To your first point, the number of stars * was intended to represent represent the significance of the test, not to label the tests. I am sorry for the confusion. You are absolutely right about the 2nd point, I will fix Commented Sep 10, 2021 at 16:33

Background and theory

In relation to what @Ben said, you could apply the gatekeeping procedure described in Bretz et al. (2009). First, fit the following regression model: $$y_i=\beta_0 + \beta_1\,\mathrm{Group}_{2, i} + \beta_2\,\mathrm{Subgroup}_{2, i} + \beta_3\,\mathrm{Group}_{2, i}\times\mathrm{Subgroup}_{2, i}$$ where $$\mathrm{Group}_{2}$$ and $$\mathrm{Subgroup}_{2}$$ are indicator variables for the second group and subgroup, respectively. According to this model, we would have three null hypotheses:

\begin{align} \operatorname{H}_{1}:&~ \beta_1 = \beta_2 = \beta_3 = 0 \\ \operatorname{H}_{2}:&~ \beta_2 = 0 \\ \operatorname{H}_{3}:&~ \beta_2 + \beta_3 = 0 \\ \end{align}

$$\operatorname{H}_{1}$$ poses that all four means are equal. $$\operatorname{H}_{2}$$ is your first hypothesis that the means of $$A$$ and $$B$$ are equal in Group 1. To see this, consider how the means for subgroup 1 ($$A$$) and 2 ($$B$$) in group 1 are parametrized in the model above: \begin{align} \mu_{1, A} &= \beta_0 + \beta_1\,0 + \beta_2\,0 + \beta_3\,0\times0 = \beta_0 \\ \mu_{1, B} &= \beta_0 + \beta_1\,0 + \beta_2\,1 + \beta_3\,0\times0 = \beta_0 + \beta_2 \end{align} So the difference between the two subgroups in group 1 corresponds to $$\beta_2$$ in the model and testing the null hypothesis that $$\beta_2 = 0$$ corresponds to testing if there is a difference between subgroups 1 and 2 within group 1.Further, $$\operatorname{H}_{3}$$ corresponds to your second hypothesis that the means of $$A$$ and $$B$$ are equal in Group 2. What's special is that we only consider $$\operatorname{H}_{2}$$ and $$\operatorname{H}_{3}$$ when $$\operatorname{H}_{1}$$ is providing evidence against the null hypothesis (i.e. $$\operatorname{H}_{1}$$ is rejected). Your third hypothesis can easily be accomodated by the procedure described here.

According to Bretz et al. (2009), we can visualize the structure among the hypohteses using the following graph:

There is only one gatekeeping hypothesis so it gets a weight of 1. We also like to control the overall familywise error rate (FWER) at $$\alpha = 0.05$$. The first hypothesis is connected to the other two via an edge with a weight of $$1/2$$. This means that if we reject $$\operatorname{H}_{1}$$ at $$\alpha$$, this local level is split equally and transferred to $$\operatorname{H}_{2}$$ and $$\operatorname{H}_{3}$$ which are then tested at $$\alpha/2$$ in this case. Furthermore, $$\operatorname{H}_{2}$$ and $$\operatorname{H}_{3}$$ are connected via edges with weight 1 which means that if $$\operatorname{H}_{2}$$ ($$\operatorname{H}_{3}$$) is rejected at its local level, this level is passed to $$\operatorname{H}_{3}$$ ($$\operatorname{H}_{2}$$).

In this case, this means that if $$\operatorname{H}_{1}$$ is rejected at $$\alpha$$, you would use the Bonferroni-Holm procedure for the two remaining hypotheses. For more complex graphs, refer to the algorithm described in Bretz et al. (2009) for calculating the adjusted $$p$$-values (see below for an R package that does this for you).

Example and implementation

The first hypothesis $$\operatorname{H}_{1}$$ can be tested using the overall $$F$$-test of the regression model described above. $$\operatorname{H}_{2}$$ is a simple $$t$$-test within the regression model and $$\operatorname{H}_{3}$$ can also be tested by an $$F$$-test. Here is an example of the procedure described above using artificial data in R. I assume that the subgroups in group $$1$$ do not differ while they do within group $$2$$. This corresponds to assuming $$\beta_2 = 0$$ in the model above (i.e. $$\operatorname{H}_{1}$$ and $$\operatorname{H}_{3}$$ are false and $$\operatorname{H}_{2}$$ is true):

library(multcomp)

# Sample size per subgroup
n <- 100

# Means and standard deviation for all four subgroups
mu_vec <- c(10, 10, 15, 25)
sigma <- 6

# Generate data
set.seed(142857)

dat <- data.frame(
y = c(rnorm(n, mu_vec[1], sigma), rnorm(n, mu_vec[2], sigma), rnorm(n, mu_vec[3], sigma), rnorm(n, mu_vec[4], sigma))
, subg = factor(rep(c("A", "B"), each = n, times = 2))
, group = factor(rep(1:2, each = 2*n))
)

# Fit regression model
mod <- lm(y~group*subg, data = dat)

# Use glht (multcomp) to calculate the p-values of all hypotheses

H1 <- matrix(c(
0, 1, 0, 0,
0, 0, 1, 0,
0, 0, 0, 1
), ncol = 4, byrow = TRUE)

H1test <- glht(mod, linfct = H1, rhs = 0)
summary(H1test, test = Ftest())

Global Test:
F DF1 DF2    Pr(>F)
1 166.4   3 396 8.891e-70

H2 <- matrix(c(
0, 0, 1, 0
), ncol = 4, byrow = TRUE)

H2test <- glht(mod, linfct = H2, rhs = 0)
summary(H2test, test = Ftest())

Global Test:
F DF1 DF2 Pr(>F)
1 0.01327   1 396 0.9083

H3 <- matrix(c(
0, 0, 1, 1
), ncol = 4, byrow = TRUE)

H3test <- glht(mod, linfct = H3, rhs = 0)
summary(H3test, test = Ftest())

Global Test:
F DF1 DF2    Pr(>F)
1 140.7   1 396 5.638e-28


Here, I used glht with contrast matrices for all hypotheses but you could use the regression output for $$\operatorname{H}_{1}$$ and $$\operatorname{H}_{2}$$. The $$p$$-values are extremely small for $$\operatorname{H}_{1}$$ and $$\operatorname{H}_{3}$$ and $$0.9083$$ for $$\operatorname{H}_{2}$$.

According to the gatekeeping procedure, we reject $$\operatorname{H}_{1}$$ at the $$0.05$$ level in the first step. In the second step, we use the Bonferroni-Holm procedure to adjust the $$p$$-values for $$\operatorname{H}_{2}$$ and $$\operatorname{H}_{3}$$, which results in adjusted $$p$$-values of $$0.9083$$ and $$1.1276\times 10^{-27}$$:

p.adjust(c(0.9083, 5.638e-28), "holm")
[1] 9.0830e-01 1.1276e-27


Hence, we reject $$\operatorname{H}_{3}$$ and fail to reject $$\operatorname{H}_{2}$$.

This can be automated using the gMCP package for R that implements the algorithms of Bretz et al. (2009). There is a graphical interface but here is the code to do the computations:

library(gMCP)

m <- rbind(H1=c(0, 0.5, 0.5),
H2=c(0, 0, 1),
H3=c(0, 1, 0))

weights <- c(1, 0, 0)

graph <- new("graphMCP", m = m, weights = weights)
pvalues <- c(8.891e-70, 0.9083, 5.638e-28)
res <- gMCP(graph, pvalues, test = "Bonferroni", alpha = 0.05)
res

P-values:
H1        H2        H3
8.891e-70 9.083e-01 5.638e-28

H1         H2         H3
8.8910e-70 9.0830e-01 1.1276e-27

Alpha: 0.05

Hypothesis rejected:
H1    H2    H3
TRUE FALSE  TRUE


Reference

Bretz, F., Maurer, W., Brannath, W., & Posch, M. (2009). A graphical approach to sequentially rejective multiple test procedures. Statistics in medicine, 28(4), 586-604.

• Thanks for the huge work. I am currently stuck at bridging the two types of hypotheses. You state that "H2 is your first hypothesis that the means of A and B are equal in Group 1". I do not follow in what way is your H2 specific to Group 1. Perhaps I misunderstand the subscript notation in the first equation. Could you please clarify Commented Sep 13, 2021 at 8:57
• @AleksejsFomins In your question, the first hypothesis states that the means of subgroups $A$ and $B$ within group 1 are equal. This corresponds to my second hypothesis because $\beta_1$ and $\beta_3$ are not considered for group 1. Commented Sep 13, 2021 at 9:24
• Ok, explanation with how different means are modeled really helped. Now it seems rather simple, but somehow was very hard to grasp from the first glance. Commented Sep 14, 2021 at 7:47

As you correctly point out, the issue here is largely a matter of dealing with the problem of multiple comparisons. In the case of nested tests, this matter is complicated by the fact that the hypotheses for the tests have direct logical implications to each other, so you are right to think that a standard application of Bonferroni's method would be problematic.

The method that is usually applied here is to first perform an over-arching test to see if there is any evidence of a difference across either subgroup. That test is not listed in your post, but it would test the hypotheses:

$$H_0: \boldsymbol{\mu}_A = \boldsymbol{\mu}_B \quad \quad \quad H_A: \boldsymbol{\mu}_A \neq \boldsymbol{\mu}_B,$$

where these vector parameters each contain the mean parameters for both subgroups. If there is no evidence of a difference then that ends the matter and the smaller tests are not performed. If there is evidence of a difference then we may then proceed to do the more specific tests 1-2. In a regression context, the overall test would be done using an F-test and the smaller tests would be done using t-tests.

• Thanks for your time :) Question 1: So F-test would be used to compare means of all 4 subgroups to look for differences? Question 2: Any advice on multiple comparisons correction strategy? Surely this problem has a standard solution Commented Sep 10, 2021 at 16:39

If you pool the observations, let's call your variable on the vertical axis y

regress y on a constant, a dummy Group (1 if observation in group 1, 0 otherwise), A (A is another dummy variable, A=1 if observation in subgroup A, 0 otherwise) and a cross effect Group*A (Group multiplied by A).

Y = Constant+ alpha Group + beta A + gamma Group*A

Constant is the mean in Group=0 (the second group), subgroup B, then:

• alpha is the test-statistic comparing the test outcomes in the two groups 1 and 2 (your third Ho),
• beta is the test comparing subgroup A and B in Group 2 (your second Ho),
• gamma is the test statistics of comparing subgroups A and B in Group 1 (your first Ho).

The F-test of the regression tests that alpha, beta, and gamma are jointly 0.

I hope this helps

• This is a very nice approach, I like it. However, it is a little unspecific. The Group*Subgroup term can tell me that there are differences within subgroups. However, I need to know whether there are differences within subgroups for each subgroup individually. So this does not answer the exact question Commented Sep 11, 2021 at 9:40
• beta tests the difference in subgroups (A and B) within group 2 while gamma tests the difference between subgroups within Group 1. So you test the difference between A nd B in both group 1 and group 2. Commented Sep 12, 2021 at 19:41
• "beta tests the difference in subgroups (A and B) within group 2" - I do not understand this statement. As is written, beta tests if there are differences between A and B within the pool of both groups, not withing group 2. What am I missing? Commented Sep 13, 2021 at 8:54
• You are right my answer is sloppy, thank you for forcing me to clarify. It is a 2X2 table each cell contains a mean, the coefficients makes you go from one cell to the next. Group=0 and A=0 meanB group 2 = constant, Group=1 A=0 MeanB group 1=cons+ALPHA; Group=0 A=1 Mean A group2=cons+BETA (second Ho BETA); Group=1 A=1 meanA group1=cons+GAMMA+BETA (First Ho GAMMA+BETA). GAMMA gives the increment for going from A group2 to A group 1 ON TOP of the difference for going from B group 2 and B group 1. GAMMA + ALPHA is the mean difference between group2 and group1 (third Ho GAMMA + ALPHA). Commented Sep 13, 2021 at 13:13
• I am sorry, it is very hard for me to follow this comment. Perhaps you can write it up nicely in the main text? Commented Sep 14, 2021 at 7:29