Finding the optimal structure of fixed / random effects in a 3 way full factorial repeated measures design with nlme/lme4

Question

I’m looking for help in finding the best / optimal structure of fixed and random effects in my repeated measures experiment data along with an explanation on why do I choose one solution over the other.

By reading various books I found a chapter on Repeated Measures designs by Andy Field (Discovering Statistics using R). He uses nlme and suggests to set up a baseline model with the following formula:

baseline <- lme(attitude ~ 1, random = ~1|participant/drink/imagery, 
                data = longAttitude, method = "ML")

from this point he advises to add fixed effects using the update function and finding the best fit (by comparing AICs or logLik).

Following his exemplary data, above formula would mean (as I understand it) a random intercept of each participant with two repeated measurement conditions (drink and imagery nested within each participant).

On the other hand, @henrik in his post Order of nested random effects in lme4, URL (version: 2017-04-13) states that

“As all participants did see all levels of your factor f1, the factor is not nested within id. A factor is nested within another factor if each instantiation of the higher order factor does not see all instantiations of the lower order factor (e.g., a factor is nested within id if id1 saw levels A and B, but not levels C and D, whereas id2 saw C & D but not levels A & B).”

Given those two examples I’m confused on how should I approach my data so that I won’t introduced any major errors while fitting my models.

My experiment was investigating the reaction times of Yes/No judgments regarding two different targets (self vs other) on two different aspects (feelings vs behavior) in two different time perspectives (recent vs distant).

Each of the 3 factors (target, aspect, time) was within subjects with fully counterbalanced order between subjects.

Also, each participant made a total of 128 judgements (on a Yes / No scale) – number of trials breaks down to 16 repetitions for every 2x2x2 (target * mode * time) combination. Each question (judgment) addressed a unique trait (happy, sad etc), but traits itself are not interesting to me.

The important part is that for every combination of my within subject factors there is a total of 16 judgments and that the depended variable is latency of the judgment controlled all 3 experimental factors + response type (was it Yes or was it No? – yes judgments are generally faster and I’d like to account for that) and trial number (responses become faster as the procedure advances and I’d like to control that too).

In previous research I used a regular GLM in SPSS to analyze similar data. Here I’ve been advised by a colleague reviewer to shift toward multilevel models (HLM, MLM, LMEM) as this can help me with:

1. Controlling for response type (Yes vs No) which would give a lot of missing cells when collapsed across all factors + response type
2. Controlling for the fact, that extremely low latencies (<200msec) are considered invalid and dropped from the analysis anyway.

Eventually I’d like to report anova-like type III F’s (or Chi2) of all main effects and their interactions along with the possibility to test for simple main effects equivalents if a 3-way interaction comes up to be significant.

To simulate data for 100 (in reality I have data for 244 subjects) participants representing a structure I will work with I came up with this code:

library(AlgDesign) #for generating a factorial design)
df <- gen.factorial(c(16,2,2,2,100), factors = "all", 
                    varNames = c("rep", "A", "B", "C", "Subject"))
df$rep <- as.numeric(df$rep)
df$Subject <- as.numeric(df$Subject)
logRT <- rnorm(n=12800, m=7, sd=1) # outcome variable
trialno <- rep(1:128, times = 100)
response <- rbinom(12800, size = 1, prob = c(0.3, 0.7)) 
df <- cbind(df, logRT, trialno, response)

Using aField's approach I should startwith a “baseline” model lme(logRT ~ 1, random = ~1|Subject/A/B/C) and work on fixed effects (which I'd like to be full factorial design):

1. Add full factorial fixed effects (ABC)
2. Add response as factor to the fixed part

On the other hand, following @Henrik post, this random structure is simply wrong.

Also I noticed that it makes a difference (for logLik) the order of nested factors - results from Subject/A/B/C differ from results of Subject/B/A/C syntax and I don't understand why and what solution is better for my experiment.

One other problem I came up is that I cannot freerly experiment with random sturctures as models are not converging. For example using my real data computing a model formula of lme(logRT ~ A * B * C + trialno, random = trialno|Subject/A/B/C) (representing a full factorial fixed effcts with random slope of trail number for every Subcjent) takes ages and I feel that it might be overspecified.

If any of the above make no sense and there is a established protocol that I might replicate and cite in the article I’d love to get a hint.

Otherwise any comments or full answers on best practices with my data structure are really more than welcome.

Answer 1

It seems that this question was partially answered in the link given in the OP comment by @Henrik. Summarising that thread:

• the "maximal" was recommended, citing Barr et al. (2013).

• there were convergence issues. It was recommended to start by removing correlations among random effects and suggests that reducing the random-effects structure is acceptable when necessary, supported by literature (eg Bates et al., 2015).

• the nested structure used by Field in nlme (random = ~1|Subject/A/B/C) is different from the random slopes approach in lme4 ((A*B*C|Subject)). The latter is now preferred for mixed models.

• simulations were recommended.

I have written about this several times here, and these are some of those threads:

Parsimonious Mixed Models

How to simplify a singular random structure when reported correlations are not near +1/-1

Should I remove correlations between random effects before removing some of them?

Specifing random effects in LMM (LME4) for a "beyond optimal model" in a top down driver approach

In a nutshell, the problem with such models is that the random structure, while perfectly reasonable from a theoretical viewpoint (if you have a variable that varies within subjects then it can be expected a priori to have a meaningful random slope). However, from a practical viewpoint it is very often the case that the data simply do not support such complex random structures, and the only way forward (staying within the mixed model framework) is the simplify the random structure. The usual place to start is by removing correlations between random effects, and if convergence problems still occur then removing random slopes.

Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of memory and language, 68(3), 255-278. https://www.sciencedirect.com/science/article/abs/pii/S0749596X12001180

Bates, D., Kliegl, R., Vasishth, S., & Baayen, H. (2015). Parsimonious Mixed Models. arXiv:1506.04967 [stat]. Abgerufen von http://arxiv.org/abs/1506.04967

Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., & Bates, D. (2017). Balancing Type I error and power in linear mixed models. Journal of Memory and Language, 94, 305–315. https://doi.org/10.1016/j.jml.2017.01.001