0
$\begingroup$

I'm analysing the data from a multi-group randomised controlled trial. Participants were assigned to one of five intervention groups, or a control group. They were tested twice, pre (T1) and post-intervention (T2). I want to examine whether any of the intervention groups showed a significantly greater improvement than the control group. Therefore I have a mixed design, with the between subjects factor Group (6 levels) and the within subjects factor Time (2 levels).

I'm planning to follow the approach outlined in Hilbert et al. (2019), of using linear mixed effects models, with contrasts for the factor Group coded to compare each intervention group to the control group like so:

contrasts(Group)

         Group1vsControl Group2vsControl Group3vsControl Group4vsControl Group5vsControl
Control                0               0               0               0               0
Group1                 1               0               0               0               0
Group2                 0               1               0               0               0
Group3                 0               0               1               0               0
Group4                 0               0               0               1               0
Group5                 0               0               0               0               1

If I were to run the following model:

example_model=lmerTest::lmer(Score~1+Time*Group+(1|Subject), data=example_data, REML=F)

then the output for the fixed effects would look like this (data randomly generated for the purposes of the example):

Fixed effects: Score ~ 1 + Time * Group 
                                Value Std.Error  DF  t-value p-value
(Intercept)                 100.96751  1.397381 294 72.25481  0.0000
TimeT2                       -0.11657  1.976195 294 -0.05899  0.9530
GroupGroup1vsControl          0.49938  1.976195 294  0.25270  0.8007
GroupGroup2vsControl         -2.36506  1.976195 294 -1.19677  0.2324
GroupGroup3vsControl         -2.81571  1.976195 294 -1.42481  0.1553
GroupGroup4vsControl         -4.17846  1.976195 294 -2.11439  0.0353
GroupGroup5vsControl         -2.43363  1.976195 294 -1.23147  0.2191
TimeT2:GroupGroup1vsControl   0.42200  2.794762 294  0.15100  0.8801
TimeT2:GroupGroup2vsControl   1.77155  2.794762 294  0.63388  0.5267
TimeT2:GroupGroup3vsControl   2.48922  2.794762 294  0.89067  0.3738
TimeT2:GroupGroup4vsControl   5.20223  2.794762 294  1.86142  0.0637
TimeT2:GroupGroup5vsControl   2.04067  2.794762 294  0.73018  0.4659

As my research question examines whether changes over time differed between groups, I am particularly interested in whether there are any significant Time*Group interactions. However, this involves comparing each intervention group to the control group (five comparisons), so therefore I need to account for the possibility of Type 1 errors by correcting for these multiple comparisons; Hilbert et al. recommend using the Benjamini-Hochberg procedure.

My question is: should I correct the p-values for all the fixed effects in the model, or just those that I am interested in (i.e. the five interaction terms)?

Reference: HILBERT, S., STADLER, M., LINDL, A., NAUMANN, F., & Bühner, M. (2019). ANALYZING LONGITUDINAL INTERVENTION STUDIES WITH LINEAR MIXED MODELS. TPM: Testing, Psychometrics, Methodology in Applied Psychology, 26(1).

$\endgroup$
0
$\begingroup$

You only need to adjust for the 5 hypotheses you are testing. This is the same concept as a scenario where there is only one treatment group being compared to a control and you include covariates such as age, gender etc. in the model. There is only one test for the treatment effect in that scenario and no adjustment is needed for multiple comparisons.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.