2
$\begingroup$

I'm working with a dataset that was collected to test the following [adapted] research question: Does listening to music (categorical x; 2 conditions: pre_music - post_music) affect test scores (y; continuous variable), and is that relationship modified by IQ level (z; continuous variable)?

A reprex dataset:

dat <- data.table(participant_id = rep(1:30, each = 2), music_condition = rep(c("pre", "post"), 30), test_score = rnorm(60, 50, 15), iq = rep(rnorm(30, 100, 15), each = 2))

Note that iq was measured only once at the beginning of the experiment and is therefore repeated twice per each participant because the value does not change between conditions.

My first instinct was to make a model, m1 wherein the effects of music_condition on test_score were tested (or, equivalently, get the p-value from a paired t-test):

lmer(test_score ~ music_condition + (1 | participant_id), data = dat) %>% 
summary()

No significant effect.

Then, to answer the second question about IQ moderating the aforementioned relation, my thought was to fit a second model, m2, with music_condition*iq interaction:

lmer(test_score ~ music_condition*iq + (1 | participant_id), data = dat) %>% 
summary()

No moderating effect of IQ.

My question: is it preferred to report the p-value from m1 to address the first research question, and the p-value from m2 to address the second research question? Or should I use both p-values (music_conditionpre and music_conditionpre:iq) from m2? Also, in the context of my research question, what does the iq fixed effects represent in m2 and shouldn’t it be exactly 1 because it does not change at all between conditions?

$\endgroup$
2
  • $\begingroup$ Do you have two test scores (one for each condition) for all participants? $\endgroup$
    – dipetkov
    Aug 7 at 22:20
  • $\begingroup$ @dipetkov in this situation, we have two test scores (outcome variable) for each participant, one test score pre-music and one post. Though, IQ score was measured once. The IQ score is repeated for each condition, because [in the context of this theoretical example] it can be assumed that IQ would not be changed between conditions. $\endgroup$
    – giopico
    Aug 8 at 18:28

2 Answers 2

1
$\begingroup$

Each person has been tested twice, once before listening to music, and a second time after listening to music. I think you want to presume that the first test does not have an influence on the second one. Although this is probably debatable (in the second test, the individual has experience with taking the test and that might improve the test result), I will make this presumption below, too. This means we have perfect matching.

In the first formula, the coefficient of music_condition will tell you if there is a total causal effect of music on the test score. So it makes sense to report the p-value of music_conditionpre for m1. You said there was no significance. However, if there were, it would also make sense to report this coefficient itself, since you probably are not only interested in whether there is an effect, but also in the size of the effect.

In the second formula, you want to check whether there is an IQ-dependent effect of music. So what you should be reporting is only the p-value of music_conditionpre:iq. . Again, in case of significance, the effect itself will be of interest, too.

Since you have used the star-operator * in the interaction term music_condition*iq, you also get an isolated iq term. This doesn't have to be zero. Imagine the reasonable scenario that the individuals with higher IQ have better test scores, which would lead to a positive coefficient for this iq term.

$\endgroup$
2
$\begingroup$

Since you have paired pre-music & post-music test scores for each participant in the study, you can analyze the data by regressing the post-music test scores (the outcome) on the pre-music scores, the IQ scores and their interaction (the predictors):

$$ \begin{aligned} \operatorname{E}\left\{ S_{\text{post}} \right\} = \beta_0 + \beta_1 S_{\text{pre}} + \beta_2 \text{IQ} + \beta_3 S_{\text{pre}} \times \text{IQ} \end{aligned} $$ where $S_{\text{pre}}$ and $S_{\text{post}}$ are the test score before and after music.

Aside: If some participants received the "treatment" (listened to music) and others didn't, we would include a treatment indicator variable. That experimental design would have been more convincing as you would have been able to compare the treated with the controls. However, it's not the design you chose, so we move forward.

How do we re-state the research question in terms of this model for $S_{\text{post}}$?

  • Does listening to music affect (increase) test scores while adjusting for IQ: Is the pre-music score coefficient $\beta_1$ significantly different from (bigger than) 1?
  • Does IQ moderate the relationship between pre- and post-music scores: Is the interaction term $\beta_3$ significant?

This analysis requires pivoting the data, so that the two scores for each participant are on the same row. There is no need for random participant effects. In R formula notation:

lm(score_post ~ score_pre * iq)
$\endgroup$
4
  • $\begingroup$ Thank you very much for the helpful comment, @dipetkov. I have a follow-up to your response, though. You mentioned that, in your revised call lm(score_post ~ score_pre * iq), the b1 would represent the effect of listening to music on test scores. However, my only point of contention is that wouldn’t b1 represent the effect of listening to music on test scores while holding iq constant? For the first research question, I am only interested in evaluating the effect of music on test scores, irrespective of IQ level. $\endgroup$
    – giopico
    Aug 8 at 21:18
  • $\begingroup$ Yes. You are right, I'll need to clarify this part. Obviously, you can consider a model without IQ altogether for an "exact translation" to your first question. But it seems to me it's better to include IQ in the analysis? This then changes the formulation of research question #1 as you point out. $\endgroup$
    – dipetkov
    Aug 8 at 21:24
  • 1
    $\begingroup$ I should also add that IQ can be splined, not need to assume that the relationship is linear. I'll update my answer accordingly. $\endgroup$
    – dipetkov
    Aug 8 at 21:28
  • $\begingroup$ I’ve never thought of including a spline in the model’s IQ term. Thanks, dipetkov. Your direction is extremely helpful! $\endgroup$
    – giopico
    Aug 9 at 3:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.