literature about permutation tests in linear mixed models to test fixed effects of categoircal variables under the null hypotheses

I'm currently analyzing a linear mixed model to model the effect of different treatments (two factors, both within and with multiple categories, e.g. treatment1_1, treatmen1_2, treatmen1_3, Treatment 2_1 ...) nested within each participant, who belong to different groups (between).

I have set up a random intercept model using lme in the nlme package, which looks roughly like this

lme(y ~ 1 + factor_within1*factor_within2*group, random = ~ 1 | ID)


where y denotes my dependent variable, factor_withinX denote the within factors (again, both categorical) and group denote the between factor (also categorical); ID is the number of each participant.

However, my dependent variable is slightly skewed and only contains values greater than 0. Thus, my residual plot looks not very convincing

Given that, I thought about using GLMM's with glmer in the lme4 package. However, using different distribution families (e.g. gamma) with different link functions (e.g. identity) did not solve the situation (I investigated the residual distribution using QQ-Plots and fit Indices like the AIC).

In a last step I thought about doing a permutation test in order to get a non-parametric alternative to test my fixed effects in the original lme model.

Three Questions:

1. Is it reasonable to go for a permutation test as a non-parametric alternative in the given situation?
2. And if yes, how is it done? Is there literature covering that topic? So far I couldn´t find anything
3. And further: does anybody know a website/paper/book, giving a example on how to implement the procedure in R?
• You wrote that your dependent variable is greater than 0, but does it also perhaps have an upper bound? If yes, then you could consider a Beta mixed effects model; see e.g.: stats.stackexchange.com/questions/370205/… . – Dimitris Rizopoulos Oct 10 '18 at 11:15
• @DimitrisRizopoulos I was under the impression that beta regression only works for (0,1), so what to do with the boundaries 0 and 1? – user2974951 Oct 10 '18 at 12:11
• @ user2974951 Indeed, one transformation that is often used to tackle this is $y^* = \{y (n - 1) + 0.5 \} / n$, where $n$ is the sample size. Otherwise, you could use something like a two-part Beta mixed model, with an extra model for the 0's and the 1's. – Dimitris Rizopoulos Oct 10 '18 at 12:48