I have a set of data and I am going to apply the lme mixed model. However, the data are not normally distributed (see the graph below).

enter image description here.

I tried (log, sqrt, zscore) and box-cox transformations and even the graphs look better, Kolmogorov-Smirnov test gave me a max p-value=0.008.

The residual plot for the lme model (used data were transformed with box-cox) is attached below and

Lilliefors (Kolmogorov-Smirnov) normality test p-value = 9.909e-12 enter image description here

To my knowledge, the residuals plot should not have any pattern and I can not see any on my plot, but why the normality tests (Kolmogorov-Smirnov / Shapiro) gave me values <0.05?

My question is:

What also can I do to normalize my data and use them for mixed model (lme)?

  • 1
    $\begingroup$ (if OP is still around ...) is the density plot based on the residuals or on the response variable? $\endgroup$ – Ben Bolker Mar 21 '18 at 2:37

A few points of general advice:

1) The plot you have of residuals vs. index isn't particularly useful for assessing the normality of the residuals. It is more useful to use a q-q plot or a histogram.

2) Don't rely on statistical tests (Shapiro-Wilk, Anderson–Darling, Kolmogorov–Smirnov, et al.) to determine if data or residuals are normally-distributed. They are sensitive to sample size. If you have a lot of data, they are likely to find a significant deviation from normal even if that deviation is small.

Here is a small example of a q-q plot and histogram in R. The chosen transformation doesn't work all that well for this example.



mtcars$mpg_trans = mtcars$mpg ^ 1.3 

model = lme(mpg_trans ~ disp + cyl, random=~1|gear,

x = residuals(model)

hist(x, prob=TRUE, col="darkgray")
Range = seq(min(x), max(x), length = length(x))
Norm = dnorm(Range, mean = mean(x), sd = sd(x))
lines(Range, Norm, col = "blue", lwd = 2)

enter image description here

qqline(x, col="red")

enter image description here

plot(predict(model), residuals(model))

enter image description here


You can try to apply the Johnson transformation to normalize a data.

# require
#Applying Johnson transformation 
# check p-value of the Anderson-Darling test

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.