2
$\begingroup$

In a LMM with interaction effect, the estimate seems unreasonable because the Score suppose to range from 0-12.

lmer(Score ~ Group * Diff * AGE + Gender + (1|participants))

Observation = 144, participants = 44

The assumptions of LMM was checked by performance package, the check_heteroscedasticity, check_normality have p-values > .05.

                    Estimate Std. Error       df t value Pr(>|t|)    
(Intercept)         -10.9589     4.8148 125.1071   -2.28  0.02454 *  
Diff                  8.5475     2.4456 109.8640    3.50  0.00068 ***
AGE                   0.2160     0.0651 127.6460    3.32  0.00119 ** 
Group1               44.1487    15.1913 132.4359    2.91  0.00429 ** 
Gender2              -0.2846     0.8334  38.2347   -0.34  0.73456    
Diff:AGE             -0.1136     0.0335 110.8691   -3.39  0.00097 ***
Diff:Group1         -21.4066     7.5641 116.2139   -2.83  0.00549 ** 
AGE:Group1           -0.6079     0.2000 132.3486   -3.04  0.00285 ** 
Diff:AGE:Group1       0.2932     0.0997 116.0751    2.94  0.00395 ** 

Not really sure whether the interaction terms can check with check_collinearity using VIF.

Low Correlation

           Term    VIF       VIF 95% CI Increased SE Tolerance Tolerance 95% CI
            AGE   2.72 [  2.17,   3.54]         1.65      0.37     [0.28, 0.46]
         Gender   1.02 [  1.00,  16.24]         1.01      0.98     [0.06, 1.00]

High Correlation

           Term    VIF       VIF 95% CI Increased SE Tolerance Tolerance 95% CI
           Diff  82.97 [ 61.56, 111.95]         9.11      0.01     [0.01, 0.02]
          Group 294.11 [217.84, 397.20]        17.15  3.40e-03     [0.00, 0.00]
       Diff:AGE  85.69 [ 63.57, 115.62]         9.26      0.01     [0.01, 0.02]
     Diff:Group 389.33 [288.32, 525.85]        19.73  2.57e-03     [0.00, 0.00]
      AGE:Group 291.55 [215.95, 393.75]        17.07  3.43e-03     [0.00, 0.00]
 Diff:AGE:Group 384.95 [285.08, 519.93]        19.62  2.60e-03     [0.00, 0.00]

EDIT

Cumulative Link Mixed Models (CLMM)

Thank you for the answers of treating the score as ordinal data, and the results is as below.

ordinal::clmm(ordered(factor(Score)) ~ Group * Diff * AGE + Gender + (1|participants))

Warning message:
(3) Model is nearly unidentifiable: large eigenvalue ratio
 - Rescale variables? 
In addition: Absolute and relative convergence criteria were met 

For the assumptions of proportional odds, it seems like there is not much consensus between different researchers.

  1. First approach (link): Use clm without random effects, and test with nominal_test, and the Diff and AGE have ps<.10 (safe enough for me).

  2. Second approach (link1, link2): Use LRT to test two models using clmm2, but the interaction effects was not described, and therefore not performed.

Nonetheless, the results and estimate is below

                    Estimate Std. Error z value Pr(>|z|)    
Diff                   8.656      2.576    3.36  0.00078 ***
AGE                    0.226      0.069    3.27  0.00106 ** 
Group                 48.673     16.772    2.90  0.00371 ** 
Gender                -0.451      0.828   -0.54  0.58597    
Diff:AGE              -0.114      0.035   -3.26  0.00110 ** 
Diff:Group           -23.459      8.429   -2.78  0.00539 ** 
AGE:Group             -0.668      0.221   -3.03  0.00244 ** 
Diff:AGE:Group         0.320      0.111    2.89  0.00388 ** 

Given that the upper and lower bound of the dependent variable is already specified in the ordered structure, it is really confusing to find that the CLMM model would have similar results out of range.

Due to the warning of model, I decided to remove the 3-way interaction, but to keep Diff * AGE despite LRT results is significant. Then, the estimates becomes much normal but still having significant effects.

                Estimate Std. Error z value Pr(>|z|)  
Diff              5.7914     2.4878    2.33    0.020 *
AGE               0.1381     0.0661    2.09    0.037 *
Group            -0.7900     0.8758   -0.90    0.367  
Gender           -0.5238     0.8109   -0.65    0.518  
Diff:AGE         -0.0713     0.0335   -2.13    0.034 *

Plot Plot2 By inspecting the plots (Age is continuous), it seems that the Estimate is too also high for Diff = 5.79. The interpretation of this variable is also not valid simply by visual inspection.

Beta distribution

Since the zero and one inflation was not available in glmmTMB, mutation was done.

mutate(score_beta = case_when(
      Score/12 == 1 ~ 1-1e-6,
      Score/12 == 0 ~ 1e-6,
      .default = totalScore/12))
glmmTMB(score_beta ~ Group * Diff * AGE + Gender + (1|participants), family=beta_family())

p > .05 for check_residuals(simulate_residuals()), but under-dispersion check_overdispersion(simulate_residuals()) was detected p = .04.

                    Estimate Std. Error z value Pr(>|z|)    
(Intercept)          -6.5910     2.3878   -2.76   0.0058 ** 
Diff                  3.5025     1.2408    2.82   0.0048 ** 
AGE                   0.0836     0.0322    2.59   0.0095 ** 
Group                31.8369     7.1772    4.44  9.2e-06 ***
Gender               -0.2793     0.4051   -0.69   0.4906    
Diff:AGE             -0.0456     0.0170   -2.68   0.0073 ** 
Diff:Group          -16.3018     3.6260   -4.50  6.9e-06 ***
AGE:Group            -0.4315     0.0952   -4.53  5.8e-06 ***
Diff:AGE:Group        0.2193     0.0481    4.56  5.2e-06 ***

This result is having even larger estimate due to the data transformation.

The difference of estimations in different approaches shows similar patterns, which match with some researches arguing that normality assumption is often not the most important concern in LMM. It could be the problem of interaction effect, but the rule of thumb is not violated for 8 fixed effects.

Knief, U., & Forstmeier, W. (2021). Violating the normality assumption may be the lesser of two evils. Behavior Research Methods, 53(6), 2576-2590. https://doi.org/10.3758/s13428-021-01587-5

$\endgroup$

2 Answers 2

1
$\begingroup$

With a limited set of ordered outcomes you need to use ordinal regression instead of least squares. Otherwise it’s all too easy to get model predictions outside the known limits, as you found. This page shows how to extend the approach to mixed models.

$\endgroup$
2
  • $\begingroup$ thank you for the suggestions, but what counts as a "limited" set of ordered outcomes? Is it an assumption for LMM model? The rule of thumb of regression models is "10 times more data than parameters", isn't that means I need 80 observations only? Marginal R2 / Conditional R2 = 0.104 / 0.702. ourcodingclub.github.io/tutorials/mixed-models statisticsbyjim.com/regression/overfitting-regression-models $\endgroup$
    – hey0god
    Apr 14 at 9:44
  • $\begingroup$ @hey0god that rule of thumb applies to trying to avoid overfitting in ordinary least squares regression. Other types of models have different "rules of thumb"; see Section 4.4 of Harrell's Regression Modeling Strategies for guidance with ordinal outcomes like yours. I said "limited" because that's what you have, but there are practical implementations of ordinal regression that can handle up to a few thousand different outcome levels--and keep all predictions within the limits of the scale. $\endgroup$
    – EdM
    Apr 14 at 14:31
1
$\begingroup$

A linear model assumes a continuous and unbounded dependent variable. So, while your results are unreasonable, I think it's because of your model, rather than the interaction.

If score is an integer, you might consider some kind of count model. If it is continuous, then you might transform it to be between 0 and 1 and then use beta regression.

In R, the GLMMTMb function in the package of the same name can handle a wide variety of models, including Poisson, truncated Poisson, negative binomial, and beta.

$\endgroup$
2
  • $\begingroup$ Can I use lmerControl to set the bound of dv? The score is integer but the residuals is normally distributed in the previous attempt, what kind of family should be used? $\endgroup$
    – hey0god
    Apr 14 at 10:25
  • $\begingroup$ I don't know. But, if it's an integer, why do this? $\endgroup$
    – Peter Flom
    Apr 14 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.