I have an experimental design with attitudes toward one positive and one negative stimulus nested within individuals. I also have a continuous predictor at the person level (a personality construct).
My plan was now to build a multi-level model with valence as level-1 predictor, (centered) personality as level-2 predictor, and the cross-level interaction of these two variables. Since I wanted to use nested chi-square statistics to assess the individual effects, the code would be something like this:

    mod0 <- lme(attitude~ 1, random = ~1|ID, data=dat, method="ML")

    mod1 <- lme(attitude ~ valence, random = ~valence|ID, data=dat, method="ML")

    mod2 <- lme(attitude ~ valence+z_personality, random = ~valence|ID, data=dat, method="ML")

    mod3 <- lme(attitude ~ valence*z_personality, random = ~valence|ID, data=dat, method="ML")

My questions are the following:

1) Is it justified to use multi-level models, given that I have only two observations per participant?

2) The random variance for valence is exaclty defined (with only two data points per person, there are no degrees of freedom left; the standard error for the random variance estimate is 0). Should I include a random effect for valence in this case?

3) I am particularly interested in the cross-level interaction (dependeing on personality, some participants are hypothesized to have a more positve attitude toward the negative stimulus than toward the positive stimulus). If I do not include the random variance for valence (see 2), this - in my understanding - means that the difference between positive and negative stimuli is the same for all participants. However, this is explicitly not what I expect. To put the question simply: Do I need to specify random variance for a level-1 predictor if I am interested in the cross-level interaction of this predictor?

For all these points the question is not "Can R / SPSS do this?" (I have tried, both can do it), but rather if I can reasonably interpret the results, given my design. Also, if you had some references for me to back this up, this would be greatly appreciated.

Thanks for your help!


I'll answer your second question first. Yes, there is no variance within the subject level in your situation, so including a random slope makes no difference when you run your analyses. In fact, in your situation, a multi-level model is exactly equivalent to a regression analysis with a difference score as the dependent variable (which, in turn, is equivalent to an ANOVA model with a 2-level within-subjects factor and an individual difference variable as a covariate that is allowed to interact with the within-subjects factor).

To demonstrate this, I have simulated some data and analyzed those data using both a multi-level model (using lmer from the lme4 package) and a regression model (using lm).


# Set the seed

# Simulate data
id <- rep(1:100, 2)
valence <- c(rep(-.5, 100), rep(.5, 100))
personality <- rep(rnorm(100, sd = 1))
eval <- .5 * valence * personality
d_long <- data.frame(id, valence, personality, eval)
d_long <- ddply(d_long, "id", mutate, 
           int = rnorm(1, sd = .5),
           val_b = rnorm(1, sd = .5),
           eval = eval + int + val_b * valence)
d_long$eval <- d_long$eval + rnorm(200, sd = .5)
d_long <- d_long[, c("id", "personality", "valence", "eval")]

# Fit the multi-level model
mod <- lmer(eval ~ valence * personality + (1 | id), data = d_long)

Linear mixed model fit by REML ['lmerMod']
Formula: eval ~ valence * personality + (valence | id) 
   Data: d_long 

REML criterion at convergence: 725.5223 

Random effects:
 Groups   Name        Variance Std.Dev. Corr
 id       (Intercept) 1.045    1.022        
          valence     1.156    1.075    0.53
 Residual             1.032    1.016        
Number of obs: 200, groups: id, 100

Fixed effects:
                    Estimate Std. Error t value
(Intercept)         -0.27116    0.12497  -2.170
valence             -0.01701    0.17949  -0.095
personality         -0.03500    0.13659  -0.256
valence:personality  0.63373    0.19617   3.231

Correlation of Fixed Effects:
            (Intr) valenc prsnlt
valence      0.262              
personality -0.029 -0.008       
vlnc:prsnlt -0.008 -0.029  0.262

Compare the results of the multi-level model to those with lm using the same data that I reshaped into so-called "wide" format so that the data would be compatible with lm:

# Reshape the data into wide format for analysis using lm
d_wide <- dcast(d_long, id + personality ~ valence, value.var = "eval")
colnames(d_wide) <- c("id", "personality", "low_eval", "high_eval")

# Predict the difference between high_eval and low_eval from personality
mod <- lm(high_eval - low_eval ~ personality, data = d_wide)

lm(formula = high_eval - low_eval ~ personality, data = d_wide)

    Min      1Q  Median      3Q     Max 
-4.7237 -1.1407 -0.1562  1.1338  5.4332 

            Estimate Std. Error t value Pr(>|t|)   
(Intercept) -0.01701    0.17948  -0.095  0.92469   
personality  0.63373    0.19617   3.231  0.00168 **
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 1.794 on 98 degrees of freedom
Multiple R-squared: 0.09624,    Adjusted R-squared: 0.08702 
F-statistic: 10.44 on 1 and 98 DF,  p-value: 0.001683 

Note that I obtained precisely the same coefficient value and $t$-value for the personality by valence interaction using lmer and lm. Although the analyses using lm that I show above do not include the main effect of personality, you can also obtain this effect through lm by predicting the average of high_eval and low_eval from personality. In general, when you only have two levels to your within-subjects variable, you can replicate the coefficient values and $t$-values from a multi-level model using a standard linear model (although this is not necessarily the case for the degrees of freedom and $p$-values; see a discussion by Doug Bates here).

This, I think, answers your second and third questions. There's nothing wrong necessarily with using a multi-level model in your situation, but you don't really gain anything in using it relative to a plain regression analysis of difference scores / averages. Likewise, there's nothing that prevents you from testing interactions between some individual difference variable and your within-subjects manipulation. What multi-level models allow (when you have more than two measurements per participant) is for random variation within multiple levels -- the interaction between the individual difference and the within-subjects manipulation is a fixed effect.

  • 1
    $\begingroup$ This is a helpful answer (+1) but be a bit careful with you coding example: 1. You need plyr to use the function ddply(), 2. you call ddply on a dataframe that doesn't exist ($d$). On a different note: while you correct say that you get the same $t$-values, you need to draw attention to the fact that defining the related degrees of freedom for that $t$-distribution is not a well defined aspect and people should not be tempted to naively extract $p$-values from it. Same $t$-values do not lead to same $p$-values in this case. (Nice answer nevertheless) $\endgroup$ – usεr11852 Jan 17 '14 at 1:38
  • $\begingroup$ You are quite right, of course. I have made corrections to both the code of my answer and have made my claims more precise (and linked to the discussion by Doug Bates about $df$s and $p$-values in multi-level models). $\endgroup$ – Patrick S. Forscher Jan 17 '14 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.