Reporting the results of a continuous moderating variable on a categorical independent variable

Question

I'm working with a dataset that was collected to test the following [adapted] research question: Does listening to music (categorical x; 2 conditions: pre_music - post_music) affect test scores (y; continuous variable), and is that relationship modified by IQ level (z; continuous variable)?

A reprex dataset:

dat <- data.table(participant_id = rep(1:30, each = 2), music_condition = rep(c("pre", "post"), 30), test_score = rnorm(60, 50, 15), iq = rep(rnorm(30, 100, 15), each = 2))

Note that iq was measured only once at the beginning of the experiment and is therefore repeated twice per each participant because the value does not change between conditions.

My first instinct was to make a model, m1 wherein the effects of music_condition on test_score were tested (or, equivalently, get the p-value from a paired t-test):

lmer(test_score ~ music_condition + (1 | participant_id), data = dat) %>% 
summary()

No significant effect.

Then, to answer the second question about IQ moderating the aforementioned relation, my thought was to fit a second model, m2, with music_condition*iq interaction:

lmer(test_score ~ music_condition*iq + (1 | participant_id), data = dat) %>% 
summary()

No moderating effect of IQ.

My question: is it preferred to report the p-value from m1 to address the first research question, and the p-value from m2 to address the second research question? Or should I use both p-values (music_conditionpre and music_conditionpre:iq) from m2? Also, in the context of my research question, what does the iq fixed effects represent in m2 and shouldn’t it be exactly 1 because it does not change at all between conditions?

Answer 1

Each person has been tested twice, once before listening to music, and a second time after listening to music. I think you want to presume that the first test does not have an influence on the second one. Although this is probably debatable (in the second test, the individual has experience with taking the test and that might improve the test result), I will make this presumption below, too. This means we have perfect matching.

In the first formula, the coefficient of music_condition will tell you if there is a total causal effect of music on the test score. So it makes sense to report the p-value of music_conditionpre for m1. You said there was no significance. However, if there were, it would also make sense to report this coefficient itself, since you probably are not only interested in whether there is an effect, but also in the size of the effect.

In the second formula, you want to check whether there is an IQ-dependent effect of music. So what you should be reporting is only the p-value of music_conditionpre:iq. . Again, in case of significance, the effect itself will be of interest, too.

Since you have used the star-operator * in the interaction term music_condition*iq, you also get an isolated iq term. This doesn't have to be zero. Imagine the reasonable scenario that the individuals with higher IQ have better test scores, which would lead to a positive coefficient for this iq term.

Answer 2

Since you have paired pre-music & post-music test scores for each participant in the study, you can analyze the data by regressing the post-music test scores (the outcome) on the pre-music scores, the IQ scores and their interaction (the predictors):

$$ \begin{aligned} \operatorname{E}\left\{ S_{\text{post}} \right\} = \beta_0 + \beta_1 S_{\text{pre}} + \beta_2 \text{IQ} + \beta_3 S_{\text{pre}} \times \text{IQ} \end{aligned} $$ where $S_{\text{pre}}$ and $S_{\text{post}}$ are the test score before and after music.

Aside: If some participants received the "treatment" (listened to music) and others didn't, we would include a treatment indicator variable. That experimental design would have been more convincing as you would have been able to compare the treated with the controls. However, it's not the design you chose, so we move forward.

How do we re-state the research question in terms of this model for $S_{\text{post}}$?

• Does listening to music affect (increase) test scores while adjusting for IQ: Is the pre-music score coefficient $\beta_1$ significantly different from (bigger than) 1?
• Does IQ moderate the relationship between pre- and post-music scores: Is the interaction term $\beta_3$ significant?

This analysis requires pivoting the data, so that the two scores for each participant are on the same row. There is no need for random participant effects. In R formula notation:

lm(score_post ~ score_pre * iq)