2
$\begingroup$

Edited to clarify further 23/04/24:

I have a fairly complex linear mixed effect model in R and I believe that there are issues to do with repeated data when running my post hoc tests. The model formula is as follows:

IFI_ALL <- 
lmer(IF_interact ~ cond_id*GoldMSI_MT*(HR + SC + ztime + IOS) +
       (1|case_id:group_id) +
       (1|stim_id) +
       (1|instrument) +
       (1|notasi),
     data= comb_df)

There are theoretical reasons why all should be in the same model, and my model fit is good. Within the dataset, the fixed effects have different levels of repetitions, whereby GoldMSI_MT has one value per participant, ztime and IOS repeat across the 3 cond_ids per participant, and HR and SC are longer time series physiological correlation data, with some missing values.

Here is an example of a portion of my dataset:

> head(comb_df[,c(1,3,5,10,17,18,19,20)])
 
A tibble: 6 × 8
  case_id ztime cond_id IF_interact IOS[,1] section    HR    SC
  <fct>   <dbl> <fct>         <dbl>   <dbl> <chr>   <dbl> <dbl>
1 1        2.59 Notated       0.837   0.533 1          NA 0.256
2 1        2.59 Notated       0.837   0.533 2          NA 0.539
3 1        2.59 Notated       0.837   0.533 3          NA 0.378
4 1        2.59 Notated       0.837   0.533 4          NA 0.148
5 1        2.59 Notated       0.837   0.533 5          NA 0.117
6 1        2.59 Notated       0.837   0.533 6          NA 0.449

I'm trying to run posthocs for the three-way interactions using emtrends.

etsSC_C <- 
    emtrends(IFI_ALL, ~ cond_id|GoldMSI_MT, 
    var =   "SC", cov.reduce=meanpm1sd, 
             infer = c(T,T,T), adjust= "BH")
  PH_SC_C<- as.data.frame(contrast(etsSC_C, "pairwise", adjust= "BH"))

GoldMSI_MT = -0.4693676:
 contrast               estimate   SE      df t.ratio p.value
 Improvised - Memorised     0.35 0.10 1391.79   3.540  0.0012
 Improvised - Notated       0.16 0.08 1387.30   2.030  0.0458
 Memorised - Notated       -0.19 0.09 1394.13  -2.000  0.0458

GoldMSI_MT =  0.3168349:
 contrast               estimate   SE      df t.ratio p.value
 Improvised - Memorised     0.04 0.07 1388.35   0.560  0.8617
 Improvised - Notated      -0.01 0.06 1386.51  -0.120  0.9075
 Memorised - Notated       -0.04 0.06 1389.47  -0.750  0.8617

GoldMSI_MT =  1.1030373:
 contrast               estimate   SE      df t.ratio p.value
 Improvised - Memorised    -0.28 0.09 1387.38  -3.150  0.0051
 Improvised - Notated      -0.18 0.08 1387.02  -2.180  0.0446
 Memorised - Notated        0.10 0.07 1387.65   1.470  0.1413

Results are averaged over the levels of: HR, SC, ztime, IOS 
Degrees-of-freedom method: kenward-roger 
P value adjustment: BH method for 3 tests

I've realised that while the contrasts for SC and HR effects are perfectly fine, the p-values when I'm running contrasts for ztime and IOS are likely to be unreliable/inflated due to the amount of duplicated rows:

etsIOS_C <- emtrends(IFI_ALL, ~ cond_id|GoldMSI_MT, var = "IOS", 
    cov.reduce=meanpm1sd, infer = c(T,T,T), adjust= "BH")
PH_IOS_C<-as.data.frame(contrast(etsIOS_C, "pairwise", adjust= "BH"))
        
        GoldMSI_MT = -0.4693676:
         contrast               estimate   SE      df t.ratio p.value
         Improvised - Memorised     0.89 0.05 1399.71  16.770  <.0001
         Improvised - Notated       0.41 0.05 1419.48   7.930  <.0001
         Memorised - Notated       -0.48 0.05 1409.22  -9.010  <.0001
        
        GoldMSI_MT =  0.3168349:
         contrast               estimate   SE      df t.ratio p.value
         Improvised - Memorised     0.37 0.04 1417.97   9.010  <.0001
         Improvised - Notated       0.03 0.03 1413.51   0.890  0.3723
         Memorised - Notated       -0.35 0.03 1413.39 -11.520  <.0001
        
        GoldMSI_MT =  1.1030373:
         contrast               estimate   SE      df t.ratio p.value
         Improvised - Memorised    -0.14 0.07 1413.82  -1.920  0.0546
         Improvised - Notated      -0.36 0.05 1414.21  -6.610  <.0001
         Memorised - Notated       -0.22 0.04 1411.25  -4.890  <.0001
    
        Results are averaged over the levels of: HR, SC, ztime, IOS 
        Degrees-of-freedom method: kenward-roger 
        P value adjustment: BH method for 3 tests

The degrees of freedom are reflecting the number of observations of the HR and SC data, not the IOS data which should be around 90.

I know this may have to do with the interactions in the original model. Are all of my assumptions here correct, and is there a solution, without using distinct() and running separate models? I've thought about looking into ref_grid and adding further arguments to cov.reduce but nothing seems to work.

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ This is not a programming request. "Are all of my assumptions here correct" is a perfectly on topic question. I am voting to leave open. $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 23 at 15:59
  • 1
    $\begingroup$ Haven't dug in too much, but with these t-statistics (abs(t) all either >4.9 or <1.92)), it's hard to imagine that adjustments to the df from 1000+ to 90 will make much difference, unless you're really interested in the exact p-values (why? ...) $\endgroup$
    – Ben Bolker
    Commented Apr 23 at 16:07
  • $\begingroup$ Thank you @BenBolker - this feels reassuring! However, I was wondering whether the t-statistics might also be influenced by these duplicated observations? I'm not necessarily interested in exact p-values, I'm just trying to determine significance, and didn't want to jump to conclusions that may be derived from flawed analyses. $\endgroup$
    – Hannah Gibbs
    Commented Apr 23 at 16:52

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.