0
$\begingroup$

I have data from a simple intervention design (n=500), where participants were measured across a number of (continuous) outcome variables at pre- and post-intervention (there was no control - not my choice). I have used lmer (from the lme4 package) to specify a mixed-effects model (one for each outcome of interest) that captures the change from pre to post (treating pre-intervention varAPRE as a predictor of post-intervention score varAPOST) whilst partialling out the contribution/variance of a number of random effects (e.g., school of the participant (1|School)).

Each model looks something like this:

model = lmer(varAPOST ~ varAPRE + (1|School), REML = FALSE )

The issue i'm having is that the coefficient for the fixed effect varAPREseems to always be positive regardless of the direction of change from pre to post intervention.

For example, the two-tired CI plot (see below) shows varA to decrease from pre to post (as hypothesised) - the mean change is -2.79. However, the model summary for the fixed effect is as follows:

 Fixed effects:
                      Estimate Std. Error        df t value Pr(>|t|)    
    (Intercept)       22.97028    1.99319 345.70193   11.52   <2e-16 ***
       varAPRE         0.49146    0.03598 463.48463   13.66   <2e-16 ***

enter image description here

Likewise, if I run a similar model for a different outcome variable, one where I expect an increase from pre to post (e.g., varB), see plot below.

model2 = lmer(varBPOST ~ varBPRE + (1|School), REML = FALSE )

I get the same coefficient direction in the output:

Fixed effects:
                   Estimate Std. Error        df t value Pr(>|t|)    
(Intercept)        27.54412    1.80938 280.17582   15.22   <2e-16 ***
     varBPRE        0.47579    0.03613 468.95373   13.17   <2e-16 ***

enter image description here

I am completely confused by this. It doesn't make any difference if I switch the order of pre and post in the model (one to the outcome, one to the predictor) or whether the pre-intervention score is centered or not. Everything in the model output looks correct, with the exception of the direction! Any help/clarity would be much appreciated.


Here are two plots varAPRE/varAPOST and varBPRE/varBPOST (coloured by school) which help to make sense of the coefficients in each model highlighted above:

enter image description here

$\endgroup$
  • $\begingroup$ The reference level is important in factors, it would appear that varAPost is the reference level in the first plot, hence the positive coefficient for Pre. I don't know what is happening in the second plot, what is Relatedness..PRE.? $\endgroup$ – user2974951 Oct 10 '18 at 9:28
  • $\begingroup$ sorry, that should be varBPRE. I've now edited it. varA and varB are continuous variables not factors. $\endgroup$ – TDUNN Oct 10 '18 at 9:33
  • $\begingroup$ Are you sure the plots are right? $\endgroup$ – user2974951 Oct 10 '18 at 9:41
  • $\begingroup$ Yes, positive. I've also calculated the mean difference by hand and it confirms the direction of the plots. Additionally, there are 12 outcome variables and each plot confirms the hypothesised direction of the intervention effect. 5 positive, 7 negative, but the coefficients are all positive in the output. $\endgroup$ – TDUNN Oct 10 '18 at 9:45
  • $\begingroup$ I may be wrong but the plots don't seem to be a good representation of your model. Your model says that a higher pre-score is associated with a higher post-score, not that the post score is lower than the pre-score. What do you see if you do plot(x = varAPRE, y = varAPOST)? (or better yet in ggplot with points colored by School). Do you get the positive association reflected in the model? $\endgroup$ – Niek Oct 10 '18 at 10:10
4
$\begingroup$

I think this should be a difference between the longitudinal and cross-sectional effects. Namely, if you would put the data in the long format, with the pre and post measurements underneath each other, and then fitted a model with time as a predictor (time = 0 for pre, and time = 1 for post), you would find the correct direction in the coefficients as in the plot.

For an illustration of this, you could have a look at slides 22-26 of my Repeated Measurements course notes.

$\endgroup$
  • $\begingroup$ Thanks for this @DimitrisRizopoulos. This is actually the model I originally specified, however someone recommended to treat pre as a predictor of post because it models the change and reduces the structure by one level - so occasion within pupil within school is reduced to pupil within school. And once pre has been centred around say the mean, the intercept gives you the predicted post-outcome for the typical person and the coefficients can be interpreted as the general rate of progress for all individuals. $\endgroup$ – TDUNN Oct 11 '18 at 6:39
  • $\begingroup$ But in this original model, was the sign of the coefficients to the direction you expected? If yes, then I mentioned in my answer above, what you're seeing is the difference between a cross-sectional and longitudinal effect. Moreover, there is a lot of discussion on whether you should treat the pre-measurement as a predictor. In the first place, you know that it has measurement error because it is an outcome. $\endgroup$ – Dimitris Rizopoulos Oct 11 '18 at 6:51
  • $\begingroup$ Yes, the sign of the coefficients in the original model was in the expected direction. Thanks for the insight re longitudinal and cross-sectional effects, I will investigate further and make a decision as what to do. $\endgroup$ – TDUNN Oct 11 '18 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.