To investigate the difference between three tasks I tested 30 individuals on each of the three tasks and ran the linear model as follows:
model <- lme(behaviour ~ task, random = ~ 1|subject, data=dat, method="ML")
With contrasts using
glht to investigate if task 1 was different from task 2 and task 3.
To furthermore investigate if individuals are consistent in their behaviour across the three different tasks I now simply test this by running three correlation tests such as
cor.test(behaviour.task1, behaviour.task2). However, I was wondering if it would not possible to get this information directly from the model, i.e. by looking at the intercept.
I have found a couple papers that have done something similar like this by reporting if the intercept is significant, however they don't explain how they performed their analyses nor can I see how showing that the intercept is significantly different from 0 in a mixed model means that individuals were consistent.
Update based on Wolfgangs answer:
First, to calculate the overall ICC of the model I ran
as.numeric(VarCorr(model)[1,1]) / (as.numeric(VarCorr(model)[1,1]) + model$sigma^2)
This shows the ICC = 0.336. To determine if the consistency of individuals is significant, I calculated if the ICC is different from 0 as follows:
model0 <- gls(behaviour ~ task, data=dat, method="ML") anova(model0, model)
The output shows:
Model df AIC BIC logLik Test L.Ratio p-value model0 1 4 1211.739 1221.648 -601.8695 model 2 5 1205.301 1217.688 -597.6505 1 vs 2 8.438081 0.0037
Thus the ICC is significantly larger than 0 (P < 0.005). Now, as I expect the tasks will differently affect the consistency of individuals, I run a new model as follows:
model2 <- gls(behaviour ~ task, correlation = corSymm(form = ~ 1 | subject), data=dat, method="ML") summary(model2)
And get the following output:
Generalized least squares fit by maximum likelihood Model: t.notcovered ~ session Data: cot.sum Subset: maleSub AIC BIC logLik 1203.922 1221.263 -594.961 Correlation Structure: General Formula: ~1 | rat Parameter estimate(s): Correlation: 1 2 2 0.233 3 0.453 0.584
thus showing that the correlation between task 1 and 2 is 0.233, between task 1 and 3 is 0.453 and between task 2 and 3 is 0.584. Now to see if this model provides a better fit than the model that assumes a single common correlation I ran the following:
anova(model, model2) Model df AIC BIC logLik Test L.Ratio p-value model 1 5 1205.301 1217.688 -597.6505 model2 2 7 1203.922 1221.264 -594.9610 1 vs 2 5.378838 0.0679
showing that the more complex model does not yield a significantly better fit, suggesting that the consistency does not differ significantly between the tasks, but as we know from the simpler model, individuals were overall consistent in their behaviour.
I added the above information with the idea that other people with a similar question may learn from this as well.