I am trying to model the progress in proficiency of task completion over time. I would like to model specifically what is the effect of completing an evaluation and receiving feedback on the ability to sufficiently complete a task. Proficiency in that task is measured on an ordinal scale that I have categorized into "sufficient" or "insufficient".
At first, I used the following mixed effects model to model this outcome:
lme4::glmer(sufficient ~ time*scale(vol_evals) + complexity + (1|task) + (1|subjectID) + (1|raterID), data = df, family = binomial("logit"), control = lme4::glmerControl(optimizer= "bobyqa", optCtrl=list(maxfun=2e5)))
sufficientis a boolean value that is
TRUEif the task is completed sufficiently.
timeis the day of the year (i.e. 1 to 365). This model is meant to model progress given feedback throughout the duration of a year.
vol_evalsis the number of evaluations/feedback received by a given
subjectIDup to the time point at which the task is being performed.
complexityis an ordinal factor (1-5) that denotes the difficulty of the task.
taskis a given task and is modeled as a random effect (there are hundreds of unique tasks).
subjectIDis a unique subject identifier. This is modeled as a random effect.
raterIDis a unique rater identifier; this is the person providing the feedback. This is modeled as a random effect (the idea being different raters rate either more harshly or more easily.
My hypothesis is that given two otherwise identical individuals completing a task (same time of the year, same task, same complexity, same rater) the individual that has received more feedback up until that point in time (modeled by
vol_evals) will be more likely to complete that task sufficiently.
The model listed above failed to converge. I therefore had to
lme4::glmer(sufficient ~ scale(time)*scale(vol_evals) + complexity + (1|task) + (1|subjectID) + (1|raterID), data = df, family = binomial("logit"), control = lme4::glmerControl(optimizer= "bobyqa", optCtrl=list(maxfun=2e5)))
The model output appears to make sense:
#> Scaled residuals: #> Min 1Q Median 3Q Max #> -8.9518 -0.3503 0.2244 0.4313 3.7796 #> #> Random effects: #> Groups Name Variance Std.Dev. #> task (Intercept) 1.1871 1.0895 #> raterID (Intercept) 2.5451 1.5953 #> subjectID (Intercept) 0.8894 0.9431 #> Number of obs: 9192, groups: task, 871; raterID, 794; subjectID, 548 #> #> Fixed effects: #> Estimate Std. Error z value Pr(>|z|) #> (Intercept) 1.72910 0.13051 13.249 < 2e-16 *** #> scale(time) 0.24813 0.05201 4.771 1.84e-06 *** #> scale(vol_evals) 0.32502 0.06046 5.376 7.61e-08 *** #> complexity.L -0.90127 0.09190 -9.807 < 2e-16 *** #> complexity.Q -0.13214 0.06343 -2.083 0.0372 * #> scale(time):scale(vol_evals) -0.08059 0.05477 -1.471 0.1412 #> --- #> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 #> #> Correlation of Fixed Effects: #> (Intr) scl(t) scl(_) cmpl.L cmpl.Q #> scale(time) -0.075 #> scl(vl_vls) 0.253 -0.421 #> complexty.L -0.232 -0.031 -0.041 #> complexty.Q 0.203 0.006 -0.008 -0.418 #> scl(tm):(_) -0.067 0.323 -0.226 -0.001 0.008
The coefficients make sense - as
number of evaluations increases, subjects are more likely to complete a task sufficiently. Likewise, as
complexity increases, subjects are less likely to complete a task sufficiently.
However, I am not sure how to interpret
scale(time). Nor am I certain this is the best way to deal with the confounding influence of
time (which is acquired by looking at the date an evaluation is filed). After all, I would expect people to improve over time regardless of if they are receiving feedback.
Is there a more appropriate way to control for the effect of time in this model?