# absence of normality of residuals - lmer or glmer?

I am in the process of analysing response time data and after inspection of the response times (as expected) they were not normally distributed, so I applied a log transformation which I know is not the optimal choice but is the solution my department tends to use. But it does not seem to work well enough after analysing the following graphs:

Are they not that bad or should I just run the analysis with Glmer? Which family is commonly recommended - ex-gaussian or the gamma distribution?

This was the process: Before log transformation

#RT model
RT.model <- lmer(RT ~ var1*var2*var3*var4+ (1|var5),data= Data, REML=FALSE)

#----------------Assumptions

# Check for normality of residuals
hist(residuals(RT.model)) #*data is skewed to the right

shapiro.test(residuals(RT.model)[0:5000]) p < .05


After log transformation

#logRT model
logRT.model <- lmer(logRT ~ var1*var2*var3*var4+ (1|var5),data= Data, REML=FALSE)

# Check for normality of residuals
hist(residuals(logRT.model)) #*data is skewed to the right

#normality of residuals - Shapiro Wilk test
shapiro.test(residuals(logRT.model)[0:5000] p < .05

• The residuals were not normal, or the pooled distribution of response times was not normal? Which caused you to want to do the log transform?
– Dave
Commented Aug 9, 2020 at 21:02
• This is all residuals of a lmer model after log transformation, not the response times. I initial did the log transformation because the RTs are obviously skewed. This approach tends to be used in my department, but I know it is not the best one.
– CatM
Commented Aug 9, 2020 at 21:05
• You don’t care about the distribution of the response times. When we make a normality assumption in linear regression, it is about the error term.
– Dave
Commented Aug 9, 2020 at 21:07
• I was not clear, I did the log transform after checking the residuals of the lmer model with the RT without any transformation, they were skewed (I was not refering to the raw data).
– CatM
Commented Aug 9, 2020 at 21:13
• You might be able to use the boxcoxmix package ... Commented Aug 10, 2020 at 1:04

It's hard to be prescriptive about these sort of situations.

If you consider gamlss() from the gamlss package in R as your model fitting function, you can consider more flexible choices for your family of distributions in addition to the ones you mentioned, such as:

1. Box-Cox Cole and Green distribution (BCCG);
2. Box-Cox Power Exponential distribution (BCPE).

The BCCG distribution allows you to simultaneously model 3 parameters of the conditional distribution of the response variable: a location parameter mu, a scale parameter sigma and a shape parameter nu. mu is the median of the distribution, sigma is approximately the coefficient of variation (for small values of sigma), and nu controls the skewness. (See the article Age- and size-related reference ranges: A case study of spirometry through childhood and adulthood by Cole et al at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2798072/ for a nice explanation of this distribution.)

BCPE has 4 parameters: a location parameter mu, a scale parameter sigma and two shape parameters: nu and tau. mu is the median of the distribution, sigma is approximately the coefficient of variation (for small sigma and moderate nu>0), nu controls the skewness and tau the kurtosis of the distribution. (Kurtosis is assumed to be absent for a BCCG distribution; BCPE is an extension of the BCCG distribution which includes kurtosis.)

The gamlss framework allows you to fit (mixed effects) models with different choices of distributions and compare the model fits using AIC or GAIC to determine which distribution is most appropriate for your data. Of course, you have to be clear on whether you are interested in modelling the median of the conditional distribution of the response (which would warrant the use of the families suggested here) or the mean. The comparison should be made across families which target the same parameter of the conditional distribution of the response.

Note: I cannot answer comments (comments do not work on my tablet), but that is why I suggested considering more flexible distributions because if you consider the deviation from normality is too pronounced, then you need something else to fall back on. In particular, you can say: considering this other, more flexible distribution, did not lead (or did lead) to a substantial improvement in the AIC or GAIC. The gamlss framework allows you to plot model diagnostics for different distributions and compare model fit performance across families - something you can't get if you consider a single family.

• But you would consider that they deviate too much from normality right?
– CatM
Commented Aug 9, 2020 at 21:30
• How does one know if the deviation from normality is too pronounced? I compared the lmer and glmer and the AIC is much higher for the glmer. npar AIC BIC logLik deviance Chisq Df Pr(>Chisq) Model 37 -40576 -40215 20325 -40650 Model.glmer 37 1494498 1494859 -747212 1494424 0 0 1
– CatM
Commented Aug 9, 2020 at 22:43
• You can examine a QQ plot which shows a 95% confidence envelope to get a sense of the seriousness of the departure from normality. The qqPlot() function in the car package would help with that. How many of the observations in your plot fall outside the envelope and/or where do they fall outside of the envelope? In your case, it looks like the peak of the distribution of residuals is more pronounced than what you’d expect from a normal, which means the tails are shorter. Commented Aug 10, 2020 at 3:54
• The Gamma distribution is nice to use, because it allows you to model the mean (not the median!) of the conditional distribution of the response variable. But it can only accomodate a certain amount of skewness, after which you may need to consider other distributions. If those distributions provide an improvement in model diagnostics, they are worth considering. However, you have to pay attention to whether they model the mean or median of the conditional distribution of the response. Commented Aug 10, 2020 at 4:03
• I was thinking about using either the gamma or the ex-guassian distribution as I have seen both being recommended for response time data. Just added the qqplot() it does seem to deviate quite substantially.
– CatM
Commented Aug 10, 2020 at 4:14