We have a question about how to choose between two non-nested mixed linear models.

The two models in R:

Model1a <- lmer(Pain ~ Visit_Year * Sleep + (Visit_Year | ID),
                data = Final_sleep_pain,  REML = FALSE) 

Model1b <- lmer(Sleep ~ Visit_Year * Pain + (Visit_Year | ID), 
                data = Final_sleep_pain,  REML = FALSE)

Both pain and sleep are time-varying variables. Pain refers to general pain intensity, ranging from 0 to 100, with a higher score meaning lower pain level;

Sleep: refers to sleep disturbance, ranging from 0 to 20, with a higher score meaning more severe sleep problem;

Visit year ranges from 0 to 11.

The data has 144233 observations and 70582 participants.

Based on the literature, there is no consistent direction of which symptom predicts which. We want to use the data-driven method, meaning let the data itself decide which model fits better, but we don’t know the criteria to decide which model is fitting better. We checked some values and showed the info in the following.

AIC, BIC, Loglikelihood, Marginal R square, Conditional R square values:

Model1a: 1298194, 11298253, -649091, 0.059, 0.58;

Model1b: 797038, 797097, -398513, 0.036, 0.62.

We are not sure which is the criteria we should look at. Also, we are not sure if we should use these criteria or some other information such as cross-validation maybe?

The reason we want to use the data-driven method is because for example if a patient complains both pain and sleep problem simultaneously, what kind of medicines should the doctors prescribe? If pain predict sleep, then pain medicine is enough; if sleep predicts pain, then sleep medications.

  • $\begingroup$ This feels like an analysis that would be better done within a structural equation modeling framework or w/ nlme - a bivariate growth model in which you model the two growth processes simultaneously. Then you can look at covariation between the pain and and sleep intercepts and similarly, the pain and sleep slopes. However the question about causal direction is one that is going to be difficult to determine given what you have described about your data. See quantdev.ssri.psu.edu/tutorials/… for an example. $\endgroup$ – Erik Ruzek Aug 3 at 19:49
  • $\begingroup$ Thank you very much for your comments!! $\endgroup$ – Yan Su Aug 5 at 3:36

This is a quite complicated question dealing with observational data and time-varying confounding (i.e., depending on the model you specify either Pain or Sleep are endogenous time-varying covariates). To get a better insight into answering you would need to used marginal structural models or joint models, depending on the assumptions you are willing to make. For both approaches there is quite an extensive literature you can study. You may find a summary regarding these issues in Chapter 12 of Diggle, Heagerty, Liang and Zeger.

Regarding your question, you cannot compare the log-likelihoods, AICs, BICs, etc. from models that have a different response variable.

  • $\begingroup$ Thank you very much Dr. Rizopoulos! Your response is very helpful! It seems maybe impossible to get the causal relationship. I agree with you about the simultaneous estimation of the sleep and pain relationship. I will discuss with my advisor and decide next steps! Thank you again! $\endgroup$ – Yan Su Aug 5 at 3:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.