Should I use multilevel modeling? One between-subjects continous predictor, one within-subjects categorical predictor

Question

I have recently collected my dissertation data. I would like to know if I should use multilevel modeling to answer my research questions.

Here is a brief summary of my method:

Participants (n = 392 women) were asked to read a passage and imagine they were in the situation described. According to the passage, the participant has just moved to a new town and hopes to make some friends; to this end, the participant has plans to meet with a woman the participant believes might become a good friend someday. Participants were then asked to indicate how they would dress if they were in the situation described in the passage. Participants chose one of several outfits which varied in modesty.

Participants then read a second passage and were again told to imagine they were in the situation described. The second passage was similar to the first, but this time, according to the passage, the participant’s social media account was recently hacked to make it seem like the participant sleeps with a lot of men, and the potential female friend with whom the participant will meet has seen the hacked account. After reading the passage, the participants followed the same procedure they did after the first passage.

In my data set, I refer to the first passage as 'control' and the second passage as 'promiscuous'.

Participants then completed a religiosity scale. Religiosity was measured with 8 items using 7-point Likert scales. The mean score for these 8 items served as a measure of participant religiosity, with higher scores indicating greater religiosity.

Therefore, I have two predictors:

1. Religiosity (Between-subjects, continuous data, ranging from 1.25 to 7.00)

2. Passage (Within-subjects, categorical [control, promiscuous])

My criterion/dependent variable is Outfit Modesty (continuous data, ranging from 2.67 to 6.41).

I would like to test the main effect of Religiosity on Outfit Modesty, the main effect of Passage on Outfit Modesty, and the interaction between Religiosity and Passage.

Note that the Outfit Modesty data are not normally distributed for either passage, as you can see below:

Also, the data distribution for Religiosity seems bimodal, as you can see below:

Can I and should I use multilevel modeling?

Answer 1

The data structure you describe is compatible with a multilevel modeling approach. Your model would take the following form:

modestyij = b0 + b1(vignette)ij + b2(religiosity)j + b3(vignette*relgiosity)ij + U0j + eij

U0j ~ N(0, sigma^2u0); eij ~ N(0, sigma^2e0)

Technically the interaction is a "cross-level" interaction, meaning that you are interacting a L1 occasion-specific variable by an L2 person-specific variable. In these cases, recent research by Heisig & Schaeffer suggests that you should allow the level 1 variable to vary across persons (often referred to as a random slope in the MLM literature). However, all subjects were exposed to the same two vignettes, so it doesn't make much sense to estimate a varying slope.

As you note, your outcome is not "normally" distributed. This is not a requirement for a linear MLM, however you will have to look at the level 1 (and level 2) residuals for your MLM to see whether they follow a normal pattern (see notes on MLM diagnostics here). If not, you may need to consider transforming your outcome scores somehow or using a generalized multilevel modeling approach.

I would recommend using MLM, however if you wanted to go in a different direction you could use one of the relevant ANOVA-type models. In your case, you might consider the random (or mixed) effects two-way ANOVA with a covariate. But the MLM is just a more parsimonious version of this model, providing arguably more information. See Robert Long's answer to this prior CV question for further information on the two approaches.