3
$\begingroup$

I am trying to model the progress in proficiency of task completion over time. I would like to model specifically what is the effect of completing an evaluation and receiving feedback on the ability to sufficiently complete a task. Proficiency in that task is measured on an ordinal scale that I have categorized into "sufficient" or "insufficient".

At first, I used the following mixed effects model to model this outcome:

lme4::glmer(sufficient ~ time*scale(vol_evals) + complexity + (1|task) + (1|subjectID) + (1|raterID), data = df, family = binomial("logit"), control = lme4::glmerControl(optimizer= "bobyqa", optCtrl=list(maxfun=2e5)))
  • sufficient is a boolean value that is TRUE if the task is completed sufficiently.
  • time is the day of the year (i.e. 1 to 365). This model is meant to model progress given feedback throughout the duration of a year.
  • vol_evals is the number of evaluations/feedback received by a given subjectID up to the time point at which the task is being performed.
  • complexity is an ordinal factor (1-5) that denotes the difficulty of the task.
  • task is a given task and is modeled as a random effect (there are hundreds of unique tasks).
  • subjectID is a unique subject identifier. This is modeled as a random effect.
  • raterID is a unique rater identifier; this is the person providing the feedback. This is modeled as a random effect (the idea being different raters rate either more harshly or more easily.

My hypothesis is that given two otherwise identical individuals completing a task (same time of the year, same task, same complexity, same rater) the individual that has received more feedback up until that point in time (modeled by vol_evals) will be more likely to complete that task sufficiently.

The model listed above failed to converge. I therefore had to scale time:

lme4::glmer(sufficient ~ scale(time)*scale(vol_evals) + complexity + (1|task) + (1|subjectID) + (1|raterID), data = df, family = binomial("logit"), control = lme4::glmerControl(optimizer= "bobyqa", optCtrl=list(maxfun=2e5)))

The model output appears to make sense:

#> Scaled residuals: 
#>     Min      1Q  Median      3Q     Max 
#> -8.9518 -0.3503  0.2244  0.4313  3.7796 
#> 
#> Random effects:
#>  Groups    Name        Variance Std.Dev.
#>  task      (Intercept) 1.1871   1.0895  
#>  raterID   (Intercept) 2.5451   1.5953  
#>  subjectID (Intercept) 0.8894   0.9431  
#> Number of obs: 9192, groups:  task, 871; raterID, 794; subjectID, 548
#> 
#> Fixed effects:
#>                              Estimate Std. Error z value Pr(>|z|)    
#> (Intercept)                   1.72910    0.13051  13.249  < 2e-16 ***
#> scale(time)                   0.24813    0.05201   4.771 1.84e-06 ***
#> scale(vol_evals)              0.32502    0.06046   5.376 7.61e-08 ***
#> complexity.L                 -0.90127    0.09190  -9.807  < 2e-16 ***
#> complexity.Q                 -0.13214    0.06343  -2.083   0.0372 *  
#> scale(time):scale(vol_evals) -0.08059    0.05477  -1.471   0.1412    
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Correlation of Fixed Effects:
#>             (Intr) scl(t) scl(_) cmpl.L cmpl.Q
#> scale(time) -0.075                            
#> scl(vl_vls)  0.253 -0.421                     
#> complexty.L -0.232 -0.031 -0.041              
#> complexty.Q  0.203  0.006 -0.008 -0.418       
#> scl(tm):(_) -0.067  0.323 -0.226 -0.001  0.008

The coefficients make sense - as time and number of evaluations increases, subjects are more likely to complete a task sufficiently. Likewise, as complexity increases, subjects are less likely to complete a task sufficiently.

However, I am not sure how to interpret scale(time). Nor am I certain this is the best way to deal with the confounding influence of time (which is acquired by looking at the date an evaluation is filed). After all, I would expect people to improve over time regardless of if they are receiving feedback.

Is there a more appropriate way to control for the effect of time in this model?

$\endgroup$
2
  • $\begingroup$ What do you mean "I am not sure how to interpret scale(time)" ? You have literally interpreted it in the preceeding paragraph. Do you mean you are unsure how to interpret that it is scaled. Also note that since it is involved in an interation, it's interpretation is conditional on vol_evals being zero. $\endgroup$ – Robert Long Oct 23 '20 at 6:30
  • $\begingroup$ How would you describe a 1 unit increase in scale(time)? I'm struggling with how - given these results and time being scaled - to describe the effect of time on likelihood of sufficiently completing a task and how this relates to vol_evals. $\endgroup$ – Dylan Russell Oct 23 '20 at 6:32
3
$\begingroup$

scale(time) standardizes time so that it has a mean of zero and a standard deviation of 1.

This means that the regression coefficient for scale(time) is the association of a 1 standard deviation change in time (unscaled) with the change in the log odds of sufficient being true, and since it is involved in an interaction with scale(vol_evals) this is conditional on scale(vol_evals) being zero.

Is there a more appropriate way to control for the effect of time in this model?

Edit: You might try to allow for a non-linear effect of time, for example by introducing a quadratic term

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.