1
$\begingroup$

I have a mixed model using factor(time) as fixed effect and subject as random effect. So the format is like this:

require(nlme)
model<-lme(outcome~factor(time), random=~time|subject)

I have 5 timepoints and I look at how the outcome changes with between each time.

My supervisor says I have to investigate the overall effect of time and only if that is significant can I state the p-values of the individual time points in the paper. Hence, my question is this: Before I am allowed to look at the changes between individual time points do I have to find a significant overall effect of time? If yes, do you just use:

anova(model)

Or will I have to compare the model to a model without time as fixed effect? I only have one fixed effect so that would be a comparison between a random model and a mixed model.

$\endgroup$
9
  • 1
    $\begingroup$ Compare a NULL model vs. a fixed effects model (time) using ANOVA to check for fixed effects significance. If time is significant add the random effects, then compare the fixed effects model with the mixed effects model to check for random efffects significance. $\endgroup$ Dec 7, 2018 at 12:49
  • $\begingroup$ Ok, will the NULL model then be: m.null<- lme(outcome~1, random=~time|subject) or lme(outcome~1,random=~1|subject)? I would guess the latter. And then only if there is significant difference between models do you continue with interpretation of the individual timepoints? $\endgroup$
    – User LG
    Dec 7, 2018 at 14:18
  • $\begingroup$ You say the null model should not contain random or fixed effect statement so can you tell me how i construct this model? Thanks $\endgroup$
    – User LG
    Dec 7, 2018 at 14:37
  • $\begingroup$ i think he means compare lm(DV~1) vs lm(DV~time) no mixed effects, just plain lm to establish significance of time. AND THEN compare lm(DV~time) VS lme(DV~time, random=time|id) to establish need for a mixed effects model. $\endgroup$
    – Huy Pham
    Dec 7, 2018 at 15:17
  • 1
    $\begingroup$ @HuyPham I agree it is weird since the assumption is violated. I don't really know what his supervisor meant by that. Maybe to compare lme(outcome~1, random=~1|subject) with lme(outcome~time, random=~1|subject). After that you could check the effect of time as a random effect too. $\endgroup$ Dec 7, 2018 at 18:42

2 Answers 2

5
$\begingroup$

A couple of points:

  • The random effects are used to model the correlations over time. Hence, if you have established that you need the random slopes, you null model will be

    model_null <- lme(outcome ~ 1, random = ~ time | subject)

  • The difference between using anova(model) versus anova(model_null, model) is that the former performs an F-test whereas the latter a likelihood ratio test. Asympotically, i.e., for large enough sample size, the two will give the same results. But they do differ when you have a small sample size. In this case, the F-test is preferred because the likelihood ratio test is anti-conservative.

  • If you want to use the F-test, it would be better to use the Kenward-Roger degrees of freedom, which are provided by the lmerTest package for linear mixed models fitted by the lmer() function of the lme4 package.

$\endgroup$
1
  • $\begingroup$ I ended up not using the random slope only the random intercept. I used the statement random =~1|subject but I used the nlme package to specify unstructured covariance structure. Reading up on that I can see that I will want to use the Kenward Roger DF but since I fitted the models with nlme to specify covariance structure I think I have to run it again with lme4 to be able to use the lmerTest package. $\endgroup$
    – User LG
    Dec 8, 2018 at 8:32
2
$\begingroup$

First off, if you treat time as categorical (i.e., factor) in the fixed effects portion of your model, shouldn't you also treat it as categorical in the random effects portion of your model? In other words, shouldn't you use something like random = ~ factor(time)|subject)?

You indicated you have 5 time points - let's call them T1, T2, T3, T4 and T5 and assume they are arranged in increasing order from the earliest time (T1) to the latest time (T5).

It seems like you have some a priori hypotheses that you are interested in testing (?). In particular, it appears that you would like to test - for the 'typical' subject - the following sets of hypotheses:

Ho: expected value of outcome at time T2 = expected value of outcome at time T1
Ha: expected value of outcome at time T2 != expected value of outcome at time T1

Ho: expected value of outcome at time T3 = expected value of outcome at time T2
Ho: expected value of outcome at time T3 != expected value of outcome at time T2

Ho: expected value of outcome at time T4 = expected value of outcome at time T3
Ho: expected value of outcome at time T4 != expected value of outcome at time T3

Ho: expected value of outcome at time T5 = expected value of outcome at time T4
Ho: expected value of outcome at time T5 != expected value of outcome at time T4

These hypotheses can be re-expressed as:

Ho: change in expected value of outcome between times T2 and T1 = 0
Ha: change in expected value of outcome between times T2 and T1 != 0

Ho: change in expected value of outcome between times T3 and T2 = 0
Ha: change in expected value of outcome between times T3 and T2 != 0

Ho: change in expected value of outcome between times T4 and T3 = 0
Ha: change in expected value of outcome between times T4 and T3 != 0

Ho: change in expected value of outcome between times T5 and T4 = 0
Ha: change in expected value of outcome between times T5 and T4 != 0

Thus, if the above sets of hypotheses are the only ones you are interested in, you would have to conduct 4 different tests. (These hypotheses only look at whether or not there is a change in the expected outcome value - for the 'typical' patient - between consecutive times. Note that != stands for "different from", Ho stands for the null hypothesis and Ha stands for the alternative hypothesis.)

Since these hypotheses are identified a priori (that is, at the study design phase and hence prior to seeing the data), you would perform them whether or not the p-value for the test of significance of the overall time effect comes out to be statistically significant. Though some people would argue that you wouldn't need to perform multiplicity adjustments for your 4 tests, I personally subscribe to the view that you do need to worry about multiplicity adjustments when it comes to the p-values you report for the 4 tests.

If it turns out that you actually don't have any a priori hypotheses, then you'd be in a situation where you have to conduct post-hoc tests to identify if there are any pairs of time points (be them consecutive or not consecutive) for which you would find a significant change in the expected outcome value. These post-hoc tests would indeed be performed only if the p-value from the test of significance of the overall time effect came out to be statistically significant.

The hypotheses for the post-hoc tests would look like this (for the 'typical' subject):

Ho: change in expected value of outcome between times T2 and T1 = 0
Ha: change in expected value of outcome between times T2 and T1 != 0

Ho: change in expected value of outcome between times T3 and T1 = 0
Ha: change in expected value of outcome between times T3 and T1 != 0

Ho: change in expected value of outcome between times T4 and T1 = 0
Ha: change in expected value of outcome between times T4 and T1 != 0

Ho: change in expected value of outcome between times T5 and T1 = 0
Ha: change in expected value of outcome between times T5 and T1 != 0

----

Ho: change in expected value of outcome between times T3 and T2 = 0
Ha: change in expected value of outcome between times T3 and T2 != 0

Ho: change in expected value of outcome between times T4 and T2 = 0
Ha: change in expected value of outcome between times T4 and T2 != 0

Ho: change in expected value of outcome between times T5 and T2 = 0
Ha: change in expected value of outcome between times T5 and T2 != 0

----

Ho: change in expected value of outcome between times T4 and T3 = 0
Ha: change in expected value of outcome between times T4 and T3 != 0

Ho: change in expected value of outcome between times T5 and T3 = 0
Ha: change in expected value of outcome between times T5 and T3 != 0

----

Ho: change in expected value of outcome between times T5 and T4 = 0
Ha: change in expected value of outcome between times T5 and T4 != 0

There are 10 post-hoc tests in all and you would need to adjust their p-values for multiplicity.

To sum up, your supervisor is right if you are in the second setting described above, where you need to conduct all possible post-hoc tests - you would only conduct these tests after finding evidence of a change in the expected outcome value for the 'typical' subject between at least two time points (be them consecutive or not). The post-hoc tests will enable you to pinpoint exactly where the changes occur.

If, however, you are in the first setting described above, where you need to conduct only a small set of a priori tests, then you would conduct those tests regardless of whether or not you find evidence of a change in the expected outcome value for the 'typical' subject between at least two time points (be them consecutive or not).

$\endgroup$
2
  • 1
    $\begingroup$ Thank you Isabella. That was a really good explanation. I did use time as factor tin all statements. since I did not know a priori where there would be a change from baseline I will only conduct the individual hypothesis-testing if there is evidence of overall effect of time. $\endgroup$
    – User LG
    Dec 8, 2018 at 8:40
  • 1
    $\begingroup$ You're welcome! Just to clarify - with a priori comparisons, you wouldn't need to know before seeing the data where the changes in the expected outcome values are (that's impossible to know anyway). All you would need to know is what pairs of time points you are interested in. $\endgroup$ Dec 8, 2018 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.