# Non significant difference between condition from LME model, when Confidence intervals clearly non overlapping

Edit with graph:

I am struggling a bit conceptually to make sense of a result I get when applying a linear mixed model to my reaction time data.

I have a 2x2 within subjects design. When I plot the data by means of an interaction plot, one of the two lines is above the other, with non-overlapping confidence intervals. However, when I apply a linear mixed-model, which looks like this:

model26 = lme(log(RT_times) ~ location*task, ~1+location*task|participant,data= data,method='REML',weights = varComb(varIdent(form=~1|location*task)),control =list(msMaxIter = 1000, msMaxEval = 1000))


I don't find any significant main effect. This is the output:

Linear mixed-effects model fit by REML
Data: data_sac

Random effects:
Formula: ~1 + task * condition | pp
Structure: General positive-definite, Log-Cholesky parametrization
StdDev    Corr
(Intercept)              0.2479765 (Intr) tskndf cndtnv
conditionvalid           0.1722409 -0.672  0.651
Residual                 0.2490666

Combination of variance functions:
Structure: Different standard deviations per stratum
Formula: ~1 | condition * task
Parameter estimates:
invalid*def     valid*def invalid*undef   valid*undef
1.0000000     0.8943147     0.8514028     0.8917650
Fixed effects:  log(latency) ~ condition * task
Correlation:
(Intr) cndtnv tskndf
conditionvalid           -0.680

Standardized Within-Group Residuals:
Min          Q1         Med          Q3         Max
-7.10755334 -0.40245682  0.02502696  0.51551241  4.18246501

Number of Observations: 5209
Number of Groups: 56


To the contrary, the p-value for task is about 0.7. I find this very strange, as for another dataset with a comparable graph, I do instead get significant results. Now, I do get that the computation of the 95% CIs and the linear mixed model are different, so they might lead to different results, but I don't get how they can be SO different. There does not seem to be anything wrong with my data, I even removed outliers etc, so it is difficult for me to grasp what is going on.

Hope the question is clear now. Many thanks for any insight you might provide!

• It's very hard to know what's going on here unless you provide details in the question about what the results of the different models are. In particular, there are really no reliable individual "main effects" when you have an interaction in the model, as the "main effect" of any one predictor will depend on how its interacting predictors are coded. Please edit the question to provide those details about the results of your models.
– EdM
May 17, 2023 at 2:05
• It helps if you provide images or tables of the data and outcome. Several issues can lead to your question 1) interpretation of overlap of error bars 2) the influence of adding a (random) effect 3) the interpretation of the p-value of a main effect or intercept May 17, 2023 at 6:06
• 4) If the plot shows confidence intervals of the levels, then the effects, when measured within the subjects, might have different accuracy. May 17, 2023 at 6:10
• @EdM thanks for your quick comments! I have edited my question and I hope that now it is clearer.
– SinC
May 17, 2023 at 8:26
• Your situation might be as "the influence of adding a (random) effect". By considering a random effect you are effectively reducing the degrees of freedom. A study based on 56 measurements or 1400 measurements, that can be a big difference in the estimates of the standard error. This difference is not visible in your confidence intervals which are computed based on the assumption that the 1400 measurements are independent. May 17, 2023 at 12:49

1. interpretation of overlap of error bars
Why is mean ± 2*SEM (95% confidence interval) overlapping, but the p-value is 0.05?

2. the influence of adding a (random) effect
Fixed vs. random effect meta regression

3. the interpretation of the p-value of a main effect or intercept

What does a significant intercept mean in ANOVA?

4. If the plot shows confidence intervals of the levels, then the effects, when measured within the subjects, might have different accuracy

In your case, I believe that it is case 2 and your graph does not correctly represent the confidence intervals.

The confidence intervals are simply those obtained as a default output from python seaborn

By considering a random effect you are effectively reducing the degrees of freedom. A study based on 56 independent groups/measurements or 1400 independent measurements, that can be a big difference in the estimates of the standard error.

This difference is not visible in your confidence intervals which are computed based on the assumption that the 1400 measurements are independent (but they are not independent, they correlate a lot within the same participant).

                         StdDev    Corr
(Intercept)              0.2479765 (Intr) tskndf cndtnv