absence of normality of residuals - lmer or glmer? I am in the process of analysing response time data and after inspection of the response times (as expected) they were not normally distributed, so I applied a log transformation which I know is not the optimal choice but is the solution my department tends to use. But it does not seem to work well enough after analysing the following graphs:




Are they not that bad or should I just run the analysis with Glmer? Which family is commonly recommended - ex-gaussian or the gamma distribution?
This was the process:
Before log transformation
#RT model
RT.model <- lmer(RT ~ var1*var2*var3*var4+ (1|var5),data= Data, REML=FALSE)

#----------------Assumptions

# Check for normality of residuals
hist(residuals(RT.model)) #*data is skewed to the right

shapiro.test(residuals(RT.model)[0:5000]) p < .05

After log transformation
#logRT model
logRT.model <- lmer(logRT ~ var1*var2*var3*var4+ (1|var5),data= Data, REML=FALSE)

# Check for normality of residuals
hist(residuals(logRT.model)) #*data is skewed to the right

#normality of residuals - Shapiro Wilk test
shapiro.test(residuals(logRT.model)[0:5000] p < .05

 A: It's hard to be prescriptive about these sort of situations.
If you consider gamlss() from the gamlss package in R as your model fitting function, you can consider more flexible choices for your family of distributions in addition to the ones you mentioned, such as:

*

*Box-Cox Cole and Green distribution (BCCG);

*Box-Cox Power Exponential distribution (BCPE).

The BCCG distribution allows you to simultaneously model 3 parameters of the conditional distribution of the response variable: a location parameter mu, a scale parameter sigma and a shape parameter nu. mu is the median of the distribution, sigma is approximately the coefficient of variation (for small values of sigma), and nu controls the skewness. (See the article Age- and size-related reference ranges: A case study of spirometry through childhood and adulthood by Cole et al at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2798072/ for a nice explanation of this distribution.)
BCPE has 4 parameters: a location parameter mu, a scale parameter sigma and two shape parameters: nu and tau. mu is the median of the distribution, sigma is approximately the coefficient of variation (for small sigma and moderate nu>0), nu controls the skewness and tau the kurtosis of the distribution. (Kurtosis is assumed to be absent for a BCCG distribution; BCPE is an extension of the BCCG distribution which includes kurtosis.)
The gamlss framework allows you to fit (mixed effects) models with different choices of distributions and compare the model fits using AIC or GAIC to determine which distribution is most appropriate for your data. Of course, you have to be clear on whether you are interested in modelling the median of the conditional distribution of the response (which would warrant the use of the families suggested here) or the mean.  The comparison should be made across families which target the same parameter of the conditional distribution of the response.
Note: I cannot answer comments (comments do not work on my tablet), but that is why I suggested considering more flexible distributions because if you consider the deviation from normality is too pronounced, then you need something else to fall back on.  In particular, you can say:  considering this other, more flexible distribution, did not lead (or did lead) to a substantial improvement in the AIC or GAIC. The gamlss framework allows you to plot model diagnostics for different distributions and compare model fit performance across families - something you can't get if you consider a single family.
