1
$\begingroup$

I hope you can help me. I performed a pre-post study with two Trainings (ViStra & LeStra), and I measured safety outcomes (e.g. Knowledge, attitudes, behavior, etc...) with questionnaires before (T1) and after (T2) the training (Time). Participants (ID) took just one of the Trainings were nested into 5 companies, and we want to know if there are differential changes in the trainings over time.

 Participants per company per training          
      V-Training  L-Training
   B  10          14
   C   9          10
   G  18          14
   H   6           8
   K  12          14
   U  31          32

Therefore, I used GLMM to evaluate safety training outcomes, like this:

Model_A<-lmer(OutcomeMean~ Time*Training + (1|company/ID), data=DataSet)

When I run the model, the first warning appears:

boundary (singular) fit: see help('isSingular')

And when I want to calculate the ICC the following message appears:

performance::icc(Model_A, by_group=TRUE) 

Can't compute random effect variances. Some variance components equal zero. Your model may suffer from singularity (see `?lme4::isSingular` and `?performance::check_singularity`).
Solution: Respecify random structure! You may also decrease the `tolerance` level to enforce the calculation of random effectn variances.

However, when I run the following model (take a look into the random effects):

Model_B<-lmer(OutcomeMean ~ Time*Training + (1|ID/company), data=DataSet)

No warning appears and it does calculate the ICC. What am I doing wrong? Participants (IDs) are assumed to be nested in companies. Is it possible that it is because within each company there are few participants?

Thank you.

Here you have the Data in Long format (ID is twice as it was tested before and after):

OutcomeMean <- as.numeric(c(3.9,4.6,3.6,4.1,4.8,4.7,4.3,5,3.5,3.8,4,4.2,4.6,4.5,3.6,4.2,3.9,4.3,3.6,3.9,4.8,4.3,3.6,4.4,4.5,4.1,4.2,3.6,4.5,4,3.6,4.4,4.3,3.6,3.9,4.2,4.3,4.6,3.5,4.6,4.1,5,3.3,4.5,"NA",5,4.4,3.5,3.6,4.1,3.7,3.9,4.2,4.1,4.1,3.7,3.2,3.4,4.3,3.8,4.6,4,4.3,3.7,4.3,3.6,3.3,3.3,4.1,4,3.4,4.6,4.4,3.1,4.9,4.5,4.1,3.6,4.4,4.5,4.5,3.3,4.4,4.3,4.2,3.1,4.3,3,4.4,"NA",3.7,3.4,4.4,4.3,2.2,5,5,5,4.3,3.6,3.6,3.8,4.2,4.9,5,3.6,4.2,5,4.3,3.8,4,5,4.3,4.5,4.7,4.6,3.9,4.4,4.5,4.9,3.7,3.3,3.8,4.6,4.9,4.7,4.8,5,"NA",3.1,4,4,4,3.6,4.8,4.8,2.3,4.3,4.2,3.8,4.7,3.6,4.1,4.3,4.4,3.5,4.2,3.4,3,4.9,4,5,4,4.2,3.1,4.6,3.9,4,5,4.2,4.4,3.6,5,4.5,4.3,4.8,4.6,4.4,3.5,4.6,3.1,3.5,2.9,5,"NA",5,3.7,5,3.5,4.6,3.4,3.4))
Training <- factor(c( "V", "V", "V", "L", "L", "V", "V", "V", "V", "L", "V", "V", "L", "V", "L", "V", "V", "L", "V", "V", "V", "V", "V", "L", "V", "V", "L", "V", "L", "V", "V", "V", "L", "V", "L", "L", "L", "L", "L", "L", "L", "V", "V", "L", "L", "L", "V", "V", "L", "L", "V", "V", "L", "V", "V", "V", "L", "V", "V", "L", "L", "L", "V", "V", "V", "L", "L", "V", "L", "L", "L", "L", "L", "V", "L", "L", "V", "V", "L", "L", "L", "L", "L", "L", "V", "V", "L", "L", "V", "L", "V", "V", "V", "V", "L", "L", "V", "V", "V", "V", "L", "V", "V", "L", "V", "L", "V", "V", "L", "V", "V", "V", "V", "V", "L", "V", "V", "L", "V", "L", "V", "V", "V", "L", "V", "L", "L", "L", "L", "L", "L", "L", "V", "V", "L", "L", "L", "V", "V", "L", "L", "V", "V", "L", "V", "V", "V", "L", "V", "V", "L", "L", "L", "V", "V", "V", "L", "L", "V", "L", "L", "L", "L", "L", "V", "L", "L", "V", "V", "L", "L", "L", "L", "L", "L", "V", "V", "L", "L", "V", "L", "V"))
company <- factor(c( "H", "H", "U", "U", "C", "U", "U", "U", "U", "U", "K", "U", "U", "G", "U", "C", "C", "G", "G", "U", "U", "U", "U", "C", "G", "C", "U", "K", "U", "U", "G", "B", "C", "U", "G", "H", "U", "U", "U", "C", "G", "G", "K", "K", "U", "G", "C", "B", "B", "B", "B", "B", "B", "B", "B", "B", "U", "U", "K", "G", "H", "H", "G", "C", "H", "G", "G", "G", "B", "U", "U", "U", "B", "K", "K", "K", "U", "K", "U", "G", "K", "U", "G", "U", "U", "U", "K", "K", "K", "C", "H", "H", "H", "U", "U", "C", "U", "U", "U", "U", "U", "K", "U", "U", "G", "U", "C", "C", "G", "G", "U", "U", "U", "U", "C", "G", "C", "U", "K", "U", "U", "G", "B", "C", "U", "G", "H", "U", "U", "U", "C", "G", "G", "K", "K", "U", "G", "C", "B", "B", "B", "B", "B", "B", "B", "B", "B", "U", "U", "K", "G", "H", "H", "G", "C", "H", "G", "G", "G", "B", "U", "U", "U", "B", "K", "K", "K", "U", "K", "U", "G", "K", "U", "G", "U", "U", "U", "K", "K", "K", "C", "H"))
Time <- factor(c( "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2", "T2"))

DataSet <- data.frame(ID, Training, company, Time, OutcomeMean)
str(DataSet)

#Remove NA
DataSet <- DataSet %>%
  filter(OutcomeMean != "NA")

#Participants per training in each company
table(DataSet$company, DataSet$Training)

#Model ID nested in company
Model_A<-lmer(OutcomeMean~ Time*Training + (1|company/ID), data=DataSet)
boundary (singular) fit: see help('isSingular')
summary(Model_A)

Here is the output of the summary(Model_A)

Linear mixed model fit by REML.
  t-tests use Satterthwaite's method [
lmerModLmerTest]
Formula: 
OutcomeMean ~ Time * Training + (1 | company/ID)
   Data: DataSet

REML criterion at convergence: 300.9

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-3.5372 -0.4916  0.0240  0.5371  2.5429 

Random effects:
 Groups     Name        Variance
 ID:company (Intercept) 0.1275  
 company    (Intercept) 0.0000  
 Residual               0.2002  
 Std.Dev.
 0.3570  
 0.0000  
 0.4475  
Number of obs: 178, groups:  
ID:company, 91; company, 6

Fixed effects:
                  Estimate Std. Error
(Intercept)        4.14532    0.08700
TimeT2             0.02066    0.09739
TrainingV         -0.15401    0.12122
TimeT2:TrainingV   0.16412    0.13487
                        df t value
(Intercept)      154.66314  47.646
TimeT2            89.00878   0.212
TrainingV        153.24518  -1.271
TimeT2:TrainingV  87.29814   1.217
                 Pr(>|t|)    
(Intercept)        <2e-16 ***
TimeT2              0.832    
TrainingV           0.206    
TimeT2:TrainingV    0.227    
---
Signif. codes:  
  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’
  0.1 ‘ ’ 1

Correlation of Fixed Effects:
            (Intr) TimeT2 TrnngV
TimeT2      -0.560              
TrainingV   -0.718  0.402       
TmT2:TrnngV  0.404 -0.722 -0.556
optimizer (nloptwrap) convergence code: 0 (OK)
boundary (singular) fit: see help('isSingular')
Improve this question
$\endgroup$
6
  • $\begingroup$ Do participants take both trainings or just one? $\endgroup$
    – frank
    Commented Oct 11, 2022 at 5:18
  • $\begingroup$ Also, in your two formulas, you have different left-hand sides, is this intentional? $\endgroup$
    – frank
    Commented Oct 11, 2022 at 5:49
  • $\begingroup$ No, Participants took just one Training Frank. And no, the companies where in the same country nut different cities. $\endgroup$
    – Estefany Rey Becerra
    Commented Oct 11, 2022 at 6:06
  • $\begingroup$ Thanks Frank, I Edit now the formulas, and the difference should be in the Random effects, which is part of my question. $\endgroup$
    – Estefany Rey Becerra
    Commented Oct 11, 2022 at 6:16
  • $\begingroup$ How do you compute the ICC? $\endgroup$
    – frank
    Commented Oct 11, 2022 at 6:17

1 Answer 1

Reset to default
0
$\begingroup$

Since each participant takes only one training, I would suggest not using the ID variable at all and doing the following: since you are interested in the difference between OutcomeMean before and after the training, create a variable for this difference, call it diff and then use the formula

diff ~ (1 | company) + (1 | Training)

This gives you the "overall" random effect of company and the "overall" random effect for Training. If you want the random effects for each combination of company and Training, use this formula:

diff ~ (1 | Training:company)
Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks @Frank, but again I had the same "boundary (singular) fit: see help('isSingular')". Could it be possible that is due to the OutcomeMean is not significant different between companies, neither within? (I just performed an ANOVA between company & Time). $\endgroup$
    – Estefany Rey Becerra
    Commented Oct 13, 2022 at 15:55
  • $\begingroup$ @EstefanyReyBecerra Did you get the error for both? $\endgroup$
    – frank
    Commented Oct 14, 2022 at 10:38
  • $\begingroup$ yes both with the same Singularity. The models run, but the variance is Zero. $\endgroup$
    – Estefany Rey Becerra
    Commented Oct 14, 2022 at 16:43
  • $\begingroup$ @EstefanyReyBecerra could you post the summary() of your fit? And what is the size of your data set? Can you maybe even post the dataset? $\endgroup$
    – frank
    Commented Oct 15, 2022 at 9:35
  • $\begingroup$ now you can find it in the main part ~ once again, thanks for your help! $\endgroup$
    – Estefany Rey Becerra
    Commented Oct 19, 2022 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.