I'm trying to interpolate the missing data point using lmer model prediction.

Subsetting to a table without any na to the missing column of interest:

table1 <- df %>% filter(!is.na(Average)) # df includes missing na value as well

Creating the model ( I have a longitudinal experiment with 4 measurements per subject and age has a contribution. I'm using age at baseline for all four meas. and am looking for the individual slope and intersection):

mod <- lmer(Average ~ time + Age + (time | subject), table1)

I'm then filling only the missing meas. with the predicted ones:

table2 <- df %>% mutate(pred = predict(mod,allow.new.levels=TRUE, .))
%>% mutate(Average = ifelse(is.na(Average), pred, Average))

The problem is, the predicted values don't seem to maintain the trend (decrease) I was expecting and in many cases even turn it into an increasing trend..

Should I use other prediction models? other interpolation methodology?


A couple of points:

  • I guess you want to use the linear mixed model to impute the missing values for the outcome Average to be used in another procedure. However, if your only purpose is to fit the mixed model, then you do not need to impute the missing data in the outcome variable. The model will provide you with correct inferences under the missing at random missing data mechanism, provided that the variance-covariance structure is correctly/flexibly specified.
  • The predict() function will give the estimated subject-specific averages at the time points you have missing data. Because these are estimated averages, you will need to account for their variance. Hence, it will be better to create multiply-imputed datasets imputing in each one the missing values from $\mathcal N \bigl ( \hat\mu_{ij}(\hat b_i), \hat \sigma^2 \bigr )$, where $\hat\mu_{ij}(\hat b_i)$ is the estimated subject-specific average provided by predict() and $\hat \sigma^2$ is the estimated variance of the error terms.
  • The discrepancies between the trends of the observed and imputed data that you observe may be attributed to the fact that the model is not correctly specified, e.g., that the longitudinal evolutions of Average are nonlinear (in your model you assume linear evolutions). Or it could also be attributed to a potential missing at random nature of the missing data. Namely, if the data are missing at random, then the observed you end up to have are a selected sample from your target population.
  • Finally, it would not be advisable to use last observation carried forward. This is a technique suffering from many shortcomings and can give your very erroneous results.
  • $\begingroup$ Thank you @Dimitris. How should create the multiply-imputed datasets you mentioned? Also, eventually, I want to know the individual slope for each subject hence my need to evaluate the missing points... $\endgroup$ – Bmorgen Dec 10 '19 at 11:51
  • $\begingroup$ If you want to get the individual slopes, you do not need to impute. You can get those from the mixed model directly using the coef() function. To do the multiple imputations as I suggested, instead of replacing the missing values with the output of the predict() function, you will need to simulate a value from a normal distribution using the rnorm() function, with mean the predicted value, and sd the residual standard error. $\endgroup$ – Dimitris Rizopoulos Dec 10 '19 at 12:13
  • $\begingroup$ would using mice for imputation then predicting the completion of the data do the work? if that so should I account for the heteroscedasticity of the subjects with 2l.norm or as whole (homoscedastic) with 2l.pan? $\endgroup$ – Bmorgen Dec 11 '19 at 17:10
  • $\begingroup$ If the measurements are taken at exactly the same time points, then you could also use MICE. But if the measurements are not taken at the same time points, then MICE would need a more elaborate approach to manage it. $\endgroup$ – Dimitris Rizopoulos Dec 11 '19 at 19:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.