Overfitting model or issue of categorical predictors?

Question

Is it possible to overfit a model by virtue of having too many categorical variables?

I have 3 categorical variables and my dependent measure is continuous (or a ratio I guess, I'm measuring accuracy/error rates).

mod4 <- lmer(accuracy ~ group + session + trialtype + 
                 session:trialtype + (1 | subject),       
             REML = FALSE, data = data)

mod5 <- lmer(accuracy ~ group + session + trialtype + 
                session:trialtype + (trialtype | subject), 
             REML = FALSE, data = data)

mod 4 I don't get any warning or error, and mod 5 I get the boundary (singular) fit message. I've included the trialtype as a random slope to replicate previous data.

When I've done an ANOVA on the two models, mod4 has lower AIC.

Edit:

I have 3 groups, 30 subjects in each group. Each subjects does a pre- and post-test task. In this task, there are 4 trial types. So I have 8 observations (4 trials, 2 session) per subject. There is complete date, nothing missing. Also, accuracy is being measured 0-1. No subjects have accuracy of 0. So they're ratios.

Answer 1

When you include the term (trialtype | subject) in your model, and given that trialtype is a categorical variable with four levels, you include four separate correlated random effects per subject (i.e., you postulate an unstructured covariance matrix for your random effects). This is a complex model with many variance components, and it's no surprise that it does not converge given the size of your dataset.

Moreover, it seems that you have a bounded outcome variable in the $(0, 1)$ interval. The linear mixed model that you fit with lmer() has normal error terms and will not respect these bounds. You could instead consider fitting a Beta mixed effects model. This is, for example, available in the GLMMadaptive package I’ve written; you can find a worked analysis here.