Nonparametric version of lme- gamm function in R

Question

Is there a non-parametric version of a linear mixed model? I've seen some people cite gamm in R as one but unsure of how I would go about using it.

In R, the code I have been using to analyze my data is:

model <- lmer(Measurement ~ Treatment * Time + (1|Family), data=mirnadata)

and this has been working great until I encountered non-parametric data (i.e., residuals are not normally distributed, which is an assumption of the lme). So essentially, I want to do the same analysis on non-parametric data but unsure of how, or even if that's even possible.

The response variable is measurement, which is a continuous variable. Treatment is a categorical variable with 2 levels, and time is a categorical variable with 3 levels. Family is my random effect, and it has 5 levels. This is the code my labmate helped me build and while it works, I don't know if it is correct:

m2 <- gamm(Measurement ~ Treatment * Time, random=list(Family=~1),data=mirnadata)
summary(m2$lme)

Answer 1

I think there is some confusion on the terms and the programming elements, so let me explain a bit what each is:

• A linear mixed model (LMM) is a linear regression technique that incorporates random effects along with the typical fixed effects. As you already noted, the residuals are assumed to be normally distributed, though they don't have to be exactly normal to work. There are different functions in R which estimate this, including lme and lme4.
• A generalized linear model (GLM) is a regression which, unlike ordinary least squares (OLS), allows one to specify exponential families (logistic, Poisson) etc. to deal with residuals that on paper shouldn't act normal. If for example we have binary data, it is very common to model this with a logistic GLM with a function like glm in R. Generalized linear mixed models (GLMMs) are just the random effects equivalent, and are fit with functions like glmmTMB or glmer rather than lmer or lme.
• Generalized additive models (GAMs) are simply GLMs that can more flexibly handle nonlinear data. If we are modeling a bivariate relationship which resembles something complex like a cosine function that can't be easily fit as such, then a GAM can be useful for finding the "true" function. Generalized additive mixed models (GAMMs) are GAMs with random effects. Both can be fit with the gam function, but the gamm4 function uses a lme4 style random effects specification, which can at times be more efficient and provide more flexibility with random effects.

Based off what you have noted here, it appears that the sole consideration you are thinking about is the residual patterns. So there are two things that you should think about to answer your question.

• What does the residual pattern look like? If the histogram/density plot of the residuals is not bell-shaped, you need to figure out why. For example, if you plot the fitted values against the residuals and find that the residuals "fan out" with increases in fitted values, this can be a sign of variance heterogeneity, and can be modeled with methods like location-scale models to deal with this. If this plot shows a very defined curve, then this is a sign of nonlinearity and is better modeled with a GAMM.
• Are there data-driven ways that your residuals look this way? Consider counts with a very limited range. They can only be positive and are often right-skewed. Modeling this with a LMM would be limited, but using a Poisson or negative binomial GLMM would potentially work better.

It may help to edit your question to include more details about the residuals, where a more precise answer can be given. I have at least covered some conceptual issues and potential avenues, but a better answer may come with better info in your question.

On a final note, the gamm specification isn't really doing anything necessarily "better" than the lmer fit. The GAMM you have specified assumes a Gaussian distribution of residuals and and has only linear terms. I would only assume you would want to use a GAMM if you are fitting something like splines (e.g. thin plate regression splines for nonlinear data).