Inverse and log function get opposite results in GLMM - which one to pick?

Question

I am trying to fit GLMM in R where we predict reaction times (RTs, dependent variable) by a continuous, uniformly distributed variable called scores (independent variable); the random effect is the participant (glmer(rt ~ scores + (1|participant)). We expect to observe a positive relationship where the increase in the predictor results in an increase in the reaction time. After fitting two GLMMs with different link functions, we get significant results but each of them is in the opposite direction. What is the correct way to pick the most suitable link function?

The reaction times data are positively skewed and according to descdist() function in R follow Gamma/lognormal distribution. I tried to fit the GLMM with 1) Gamma distribution with log link function and 2) Gamma distribution with inverse link function. Both results are significant but in opposite directions. Specifically, the result from the GLMM with gamma distribution and inverse link function corresponds to our hypothesis. But I am not completely sure how to justify the choice of the inverse link function.

I have come across the same trend reported here. Based on the explanation, each link function links the predictor to the response variable in different ways. But it is still not clear to me how to decide which one of them is more suitable for our question. How do I choose the right link function for my design? Is there a formal way to decide this?

Answer 1

As you have recognized, the difference in the coefficients is related to the link functions. The inverse link functions results in an inverse function of the linear predictor and therefore leads to a negative coefficient when the relationship between dependent and independent variable is positive.

To decide which model family to use, I would suggest using the DHARMa package to see which model fits best. It has a great vignette which will guide you through residual diagnostics for generalized linear mixed models.

You can also compare the residual deviance of your models with anova(m1, m2) and their parsimony with AIC(m1, m2) (although anova will also include AIC and BIC, which are indicators of residual deviance weighted by model complexity, lower values indicating lower residual deviance and/or simpler models, which are preferable). However, this part should not supplant residual diagnostics which are essential to see if the models fit the data.