3
$\begingroup$

In my longitudinal data, I firstly build a model with two fixed effects, session.week and sync. The former one is just the time variable.

I actually have two subquestions :

(1) What if time variable is not significant? can I remove it?

When I run the following code with R

  quality.Model.1<- lmer(quality ~  session.week + sync  + (1|group.name/student.id), data = dfSubset)
  pvals.fnc(quality.Model.1)

What I get is the following

  $fixed
                Estimate MCMCmean HPD95lower HPD95upper  pMCMC Pr(>|t|)
  (Intercept)     3.3166   2.9966     0.9498     4.6601 0.0154   0.0000
  session.week   -0.0458  -0.0265    -0.2369     0.1867 0.7934   0.6352
  sync            1.3079   1.6866     0.1026     3.7183 0.0244   0.0206

  $random
                   Groups        Name Std.Dev. MCMCmedian MCMCmean HPD95lower HPD95upper
  1 student.id:group.name (Intercept)   0.3303     0.1417   0.1535     0.0000     0.3912
  2            group.name (Intercept)   0.0000     0.1975   0.3523     0.0000     1.2247
  3              Residual               0.6365     0.6798   0.6859     0.5408     0.8552

As you see, the time variable is not significant, but sync is significant (p<.05) If I run a summary of the above model, we got:

  AIC   BIC logLik deviance REMLdev
 113.3 124.4 -50.67    96.63   101.3

Actually although this is a longitudinal study, I don't actually expect the time would change my independent variable. The non-significance result actually confirms my expectation. Since the time variable (session.week) is not significant, can I remove it in my model, so it becomes:

 quality.Model.2<- lmer(quality ~  sync  + (1|group.name/student.id), data = dfSubset)
 pvals.fnc(quality.Model.2)

The result of this models is :

  $fixed
                Estimate MCMCmean HPD95lower HPD95upper  pMCMC Pr(>|t|)
  (Intercept)      3.091    2.897     1.5119      3.931 0.0032   0.0000
           sync    1.413    1.691     0.3116      3.320 0.0092   0.0074

  $random
                   Groups        Name Std.Dev. MCMCmedian MCMCmean HPD95lower HPD95upper
  1 student.id:group.name (Intercept)   0.3371     0.1480   0.1583     0.0000     0.3903
  2            group.name (Intercept)   0.0000     0.1787   0.2955     0.0000     1.0023
  3              Residual               0.6284     0.6721   0.6784     0.5373     0.8353

And if we summary the result, we got:

  AIC BIC logLik deviance REMLdev
  108.7 118 -49.35    96.91   98.71

We don't lose the significance of sync, but the AIC has reduced. Can I conclude that my second model (without the time variable) is better? And can I just remove the time variable safely? It just made me feel wired to remove the time variable in a longitudinal study.

(2) What if the the significance of sync is canceled when testing a model with interaction terms?

Suppose we don't give up time variable, and I built a mixed model with an interaction term as follows:

  quality.Model.3<- lmer(quality ~  session.week * sync  + (1|group.name/student.id), data = dfSubset)
  pvals.fnc(quality.Model.3)

and we got the following :

  $fixed
                             Estimate MCMCmean HPD95lower HPD95upper  pMCMC Pr(>|t|)
  (Intercept)                  4.6887   4.3486      1.058      7.477 0.0200   0.0004
  session.week                -0.3859  -0.3402     -0.953      0.313 0.2842   0.1744
  sync                        -0.6694  -0.2381     -4.267      3.874 0.8706   0.6835
  session.week:sync            0.5110   0.4610     -0.395      1.368 0.2916   0.2018

  $random
                   Groups        Name Std.Dev. MCMCmedian MCMCmean HPD95lower HPD95upper
  1 student.id:group.name (Intercept)   0.3488     0.1465   0.1576       0.00     0.3941
  2            group.name (Intercept)   0.0000     0.1861   0.3387       0.00     1.2614
  3              Residual               0.6260     0.6808   0.6869       0.53     0.8444

Nothing is significant at all, including the interaction term!!!, after introducing the interaction term. If we do summary on it:

     AIC   BIC logLik deviance REMLdev
     113.7 126.7 -49.85    94.92   99.71

The AIC is larger than both quality.Model.1 and quality.Model.2.

(3)What to choose

So, which model should I choose if I want to see the effect of sync on quality? If I remove the time variable as fixed effects, then it's done perfectly. If I don't remove the time variable, then the main effects model suggests the significance of sync, but the interaction model cancels out everything.

$\endgroup$
4
$\begingroup$

I don't think the fact that this is longitudinal data changes any of the usual points about this sort of analysis:

1) If you have an interaction term in your model, you want all the main effects involved in the interaction in your model (with very rare exceptions)

2) We shouldn't choose variables exclusively by whether they are significant - they can also be important for other reasons. Here, the time variable is important because a) It's in an interaction b) It's "natural" - you sort of have to look at the effect of time in a longitudinal analysis c) It affects other parameters.

3) We also shouldn't reject models just because they don't fit our expectations. In your case, it looks like there is an interaction of time and sync.

4) Once there is an interaction, analyzing the main effects is complex. So your statement that the interaction cancels out everything is, while true looking at the output, not really so. The main effects posted when an interaction is included are the effects of that variable when the other variable = 0. So.... you are seeing the effect of time when sync = 0 and the effect of sync when time = 0.

$\endgroup$
  • $\begingroup$ Thanks Peter. I have two comments : (1)Some one suggests to select the best model based on AIC and BIC, if this is the case, then definitely, the model without time is the best. (2) After introducing the interaction terms, nothing is significant, even the interaction term is not significant. But the main effect is canceled out. In this case (interaction term is not significant), how do you think I should report the result? Should I report the variable sync significantly affects quality? Or, nothing affects quality significantly? $\endgroup$ – nan Sep 23 '13 at 9:55
  • $\begingroup$ 1) Using AIC or BIC is OK, as long as the model you wind up choosing doesn't violate rules. Including an interaction without the main effects does violate a rule. 2) Whether to include the interaction is up to you - is it interesting? Is it useful? Does it help you understand what is going on? 3) However, deleting time seems to me to be contrary to the spirit of longitudinal analysis. Would you include it only as a random effect? That might be OK. $\endgroup$ – Peter Flom - Reinstate Monica Sep 23 '13 at 10:18
  • 1
    $\begingroup$ (1)I don't get why you say "including an interaction without the main effects". For the three models i built, I never include an interaction without the main effects. The model without time is without interaction terms as well. (2) the design is longitudinal because it is on a course, and we did measurements 4 times during a semester, but our main aim is not to examine the effect of time, but to test the effect of the variable "sync". You think it also make sense to do "quality ~ sync + (1|group.name/student.id) + (1|session.week)", where the time variable is included as a random effect? $\endgroup$ – nan Sep 23 '13 at 10:30
  • $\begingroup$ and in R, session.week * sync means session.week + sync + session.week:sync. So there is no problem of including an interaction with the main effects. $\endgroup$ – nan Sep 23 '13 at 11:11
  • $\begingroup$ I don't see model selection as useful here, as opposed to fitting a well-formed full model and interpreting it. $\endgroup$ – Frank Harrell Sep 23 '13 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.