# Diagnosing why MICE fails to impute data with either pmm or passive imputation

I'm having the worst time getting mice (version 3.3.0 under R 3.4.4 in debian stretch) to impute missing values in a particular dataset. This dataset describe a scale development effort with planned missingness in the variables that are available. Essentially, columns 11:25 are the initial versions of the scales, 54:62 are intermediate versions of some of them, and 140:165 are the final versions.

I want to use predictive mean matching to impute values in columns 140:165 for participants who may not have the final versions of each scale. They're continuous measures, so pmm seems to be appropriate in this case. Columns 257:260 are factor scores for the final version; because those scores are just linear combinations of the scale scores, I want to use passive imputation to impute factor scores.

When I open the dataset and run the commands below, the final 10 imputation acts as if it's imputing data, as it takes over a minute to execute. However, when I look at anon_10imp_scales$imp, it's filled with NA, suggesting it didn't actually write any imputed data. plot(anon_10imp_scales) also fails, so it seems clear that no imputed data are being written to the object. I used an initial imputation to seed the predictorMatrix and method. I then use a bit of coding to generate the matrix and method I want so that it can be reproduced easily. After the imputation, I've looked at anon_10imp_scales$formulas, and the formulas seem to be correct for the imputations I want to use for the variables I want imputed. Likewise, anon_10imp_scales$method seems to have the correct values all the way across (though you may note a line in the code that deletes an errant value at the end of the method that's generated after I zero everything out). ################### # Import packages # ################### library(mice) #Basic imputation library(tictoc) #Timing execution ############# # Load data # ############# load("anonScales.Rdata") ########################### # CREATE IMPUTED DATASETS # ########################### ################################ # 1. Impute all scales we want # ################################ # Create list of scales whose values should be imputed scaleSet <- subset(anonScales, select=c("var140", "var141", "var142", "var143", "var144", "var145", "var146", "var147", "var148", "var149", "var150", "var151", "var152", "var153", "var154", "var155", "var156", "var157", "var158", "var159", "var160", "var161", "var162", "var163", "var164", "var165")) # Get names of scales scaleNames <- dimnames(scaleSet)[[2]] ### Generate imputation parameters ### # Matrix for imputation generated by: ini <- mice(anonScales, maxit=0, visitSequence="monotone") ## Create predictor matrix ## anonPred <- as.data.frame(ini$predictorMatrix)

# Zero out all column variables that should never be used as predictors
anonPred[,1:10]    <- 0
anonPred[,26:53]   <- 0
anonPred[,63:139]  <- 0
anonPred[,166:261] <- 0
# Zero out all row variables that should never be predicted
anonPred[1:139,]   <- 0
anonPred[166:257,] <- 0
# Force prediction of var259; the algorithm says it's collinear, but it shouldn't be
anonPred[259,11:25]   <- 1
anonPred[259,54:62]   <- 1
anonPred[259,140:165] <- 1

# Revert anonPred to matrix form
anonPred <- as.matrix(anonPred)

## Create imputation method ##

# Seed method variable
anonMethod <-ini$method # Blank out all entries to default to not imputing anonMethod[1:dim(anonScales)[[2]]] <- "" # Put pmm entries in all scales that should be imputed for (index in 1:dim(anonScales)[[2]]) { eval(parse(text=paste0("anonMethod[\"", scaleNames[index], "\"] <- \"pmm\""))) } # Get rid of extra variable?! anonMethod <- anonMethod[-(index+1)] # Passively impute factor scores anonMethod["var258"] <- "~I((5-var149+var151+var146+var140+var145+var150+var148+var144+var141+var142+var143+var147)/12)" anonMethod["var259"] <- "~I((5-var152+5-var160+5-var159+5-var157+var155+var158+5-var156+5-var154+var153)/9)" anonMethod["var260"] <- "~I((5-var164+5-var165+var163+var161+5-var162)/5)" anonMethod["var261"] <- "~I((5-var149+var151+var146+var140+var145+var150+var148+var144+var141+var142+var143+var147+ 5-var152+5-var160+5-var159+5-var157+var155+var158+5-var156+5-var154+var153+ 5-var164+5-var165+var163+var161+5-var162)/26)" ### Impute data ### # Make 10 imputations tic(msg="10 imputations", quiet=FALSE) anon_10imp_scales <- mice(data=anonScales, m=10, maxit=30, method=anonMethod, visitSequence="monotone", predictorMatrix=anonPred, seed=20190201) toc()  In debugging this, I realized that some of the scales have been finalized since our first round of data collection, so they are identical to the final columns I'm trying to impute. The same behavior occurs if I attempt imputing only the scales I know are unique. ############################################### # 2. Impute only scales that are missing data # ############################################### # Create list of scales whose values should be imputed scaleSet <- subset(anonScales, select=c("var140", "var143", "var144", "var148", "var149", "var150", "var151", "var153", "var155", "var156", "var159", "var161", "var162", "var163", "var164", "var165")) # Get names of scales scaleNames <- dimnames(scaleSet)[[2]] ### Generate imputation parameters ### # Matrix for imputation generated by: ini <- mice(anonScales, maxit=0, visitSequence="monotone") ## Create predictor matrix ## anonPred <- as.data.frame(ini$predictorMatrix)

# Zero out all column variables that should never be used as predictors
anonPred[,1:10]    <- 0
anonPred[,26:53]   <- 0
anonPred[,63:139]  <- 0
anonPred[,166:261] <- 0
# Zero out all row variables that should never be predicted
anonPred[1:139,]   <- 0
anonPred[166:257,] <- 0
# Force prediction of var259; the algorithm says it's collinear, but it shouldn't be
anonPred[259,11:25]   <- 1
anonPred[259,54:62]   <- 1
anonPred[259,140]     <- 1
anonPred[259,143]     <- 1
anonPred[259,144]     <- 1
anonPred[259,148:151] <- 1
anonPred[259,153]     <- 1
anonPred[259,155]     <- 1
anonPred[259,156]     <- 1
anonPred[259,159]     <- 1
anonPred[259,161:165] <- 1

# Revert anonPred to matrix form
anonPred <- as.matrix(anonPred)

## Create imputation method ##

# Seed method variable
anonMethod <-ini\$method

# Blank out all entries to default to not imputing
anonMethod[1:dim(anonScales)[[2]]] <- ""

# Put pmm entries in all scales that should be imputed
for (index in 1:dim(anonScales)[[2]]) {
eval(parse(text=paste0("anonMethod[\"", scaleNames[index], "\"] <- \"pmm\"")))
}

# Get rid of extra variable?!
anonMethod <- anonMethod[-(index+1)]

# Passively impute factor scores
anonMethod["var258"] <- "~I((5-var149+var151+var146+var140+var145+var150+var148+var144+var141+var142+var143+var147)/12)"
anonMethod["var259"] <- "~I((5-var152+5-var160+5-var159+5-var157+var155+var158+5-var156+5-var154+var153)/9)"
anonMethod["var260"] <- "~I((5-var164+5-var165+var163+var161+5-var162)/5)"
anonMethod["var261"] <- "~I((5-var149+var151+var146+var140+var145+var150+var148+var144+var141+var142+var143+var147+
5-var152+5-var160+5-var159+5-var157+var155+var158+5-var156+5-var154+var153+
5-var164+5-var165+var163+var161+5-var162)/26)"

### Impute data ###
# Make 10 imputations
tic(msg="10 imputations", quiet=FALSE)
anon_10imp_scales <- mice(data=anonScales,
m=10,
maxit=30,
method=anonMethod,
visitSequence="monotone",
predictorMatrix=anonPred,
seed=20190201)
toc()


Ideally, I'd like to impute values for our criterion scales as well, but when I've tried doing that, the values fluctuate wildly such that they aren't really usable (perhaps because about 2/3 of the scores are missing, even though they should be missing completely at random). Thus, I'm trying to get the basic imputation problem solved before tackling the larger imputation issue.

• This might be too involved for CV and may require the help of a consultant. – Noah Mar 6 '19 at 6:13