I have a problem with interpreting Reaction Time results with mixed-effect models.
In the experiment, participants were split into 2 conditions. They looked at the same set of pictures and then took part in a Reaction Time task. The data is unbalanced, because I removed outliers according to a pre-specified procedure (I'm aware of the issues with outlier removal). So x participants in Condition 1 and y participants in Condition 2 provided Reaction Times to the same pictures; each person only once, but for some participants not all responses are available.
If to follow the simplest standard procedure, I could aggregate the results by participants (i.e. get participant means) and compare the groups with a t-test. This gives a significant difference.
But there are additional factors I want to consider (for example ProceedingRT and Trial Sequence, which were shown to correlate with Reaction Times in the task: Baayen & Milin 2010). Plus, there is the independence assumption that would be violated (since the same participant is more likely to provide similar answers due to his/her reaction skills etc.). So linear mixed-effects model seems much more adequate.
I will also add gender to the mix, for theoretically-justified reasons. The most obvious model in this case would be:
RT(inverted) ~ cond + gender + RTtrial + proceedingRT + (1|ids) + (1|pic)
('ids' stands for participant id; 'pic' stands for 'picture'). I also inverted RT (1/RT) and standardised it for easier interpretation. This results in:
Random effects:
Groups Name Variance Std.Dev.
ids (Intercept) 0.25276 0.5028
pic (Intercept) 0.05411 0.2326
Residual 0.54659 0.7393
Number of obs: 309, groups: ids, 31; pic, 14
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) -0.17437 0.22172 28.64000 -0.786 0.43805
cond2 -0.27471 0.20965 24.92000 -1.310 0.20204
genderm 0.40161 0.21222 25.20000 1.892 0.06998 .
RTtrial -0.21375 0.04472 276.97000 -4.780 0.00000285 ***
proceedingRT -0.15225 0.05542 292.20000 -2.747 0.00638 **
Marginal R^2 was 0.15 and Conditional R^2 was 0.45.
P-values are just a visual helper from lmerTest
(I'm aware of the issues with them, too:). Anyway, t-value for cond
is pretty low and not well-justified if I compare it against a model with no cond
as a predictor. However, when I run the same model without the random by-participant effect:
RT(inverted) ~ cond + gender + RTtrial + proceedingRT + (1|pic)
the effect of cond
rises.
Random effects:
Groups Name Variance Std.Dev.
pic (Intercept) 0.0466 0.2159
Residual 0.7628 0.8734
Number of obs: 309, groups: pic, 14
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) -0.16677 0.12121 77.93000 -1.376 0.172805
cond2 -0.24750 0.10322 293.55000 -2.398 0.017116 *
genderm 0.39617 0.10646 295.78000 3.721 0.000237 ***
RTtrial -0.24894 0.05141 301.91000 -4.842 0.00000205 ***
proceedingRT -0.27026 0.06032 299.17000 -4.480 0.00001063 ***
Marginal R^2 was 0.2 and Conditional R^2 was 0.25.
AIC value of the first model (with (1|ids)
) is lower. However, the second model, despite performing worse overall, has a higher Marginal R-squared, meaning it explains bigger proportion of variance by fixed effects than the first model.
I generally understand that the first model can account for the random by-person variance, therefore accounts for more theoretically-justified variance, and therefore performs better overall. What I don't understand is why the random effect 'steals' the explained variance from the fixed effect. My understanding so far was that it should be the opposite: that random effects are only used to show how much 'leftover' variance there is in the data after accounting for all significant fixed effects. After all, I want to maximise Marginal R^2 - I want to know how well I can predict a given phenomenon with my known, well-defined, quantified, fixed factors.
I am not sure how to interpret this result: the impact of condition is significant when the by-participant variance is ignored, but it is not, when we account for its random effect... Does this mean that participants got split into conditions in some funny, regular way that elevated the cond
effect?
Or perhaps I'm doing something wrong by including both cond
and (1|ids)
in the model, even though each participant could only be in one condition? If so, how could I account for by-participant variance within each condition separately?
And just to make it clear: I'm absolutely fine with either effect of condition
, I just want to understand where this behaviour comes from. And why the result is contrary to the simplest t-test?
---ADDITIONAL INFO about the variables
RTtrial is a number at which the picture was seen in the sequence. There were 14 pictures, the order was random, and there were 14 additional 'distractors' which are completely removed from analysis. In total there were 28 trials but I'm looking at only half of them. So RTtrial is a random set of 14 numbers from the range 1-28, and the numbers are unique within participant. Everyone saw the same pictures, that's why I want a random by-picture effect (some pictures might be easier to react to than others).
---OUTPUT from optinfo requested in comments (I'm not sure how to interpret it)
mem2@optinfo
$optimizer
[1] "bobyqa"
$control
$control$iprint
[1] 0
$derivs
$derivs$gradient
[1] -0.000002817160 0.000003574314
$derivs$Hessian
[,1] [,2]
[1,] 128.05035 -10.96187
[2,] -10.96187 244.71216
$conv
$conv$opt
[1] 0
$conv$lme4
list()
$feval
[1] 50
$warnings
list()
$val
[1] 0.6800193 0.3146293