Does MICE work with 100% correlated missing values?

Question

I have a dataset with missing values which I would like to impute by using Multiple Imputation by Chained Equations (MICE). The important characteristics of the dataset is that, for the columns containing missing values, a household either lacks all the values or has all of them. I give a toy dataset with the same feature below: (only that my data has 36 columns, of which 18 contains missing variables, and 6119 rows, of which 1529 contains missing variables)

a=c(12,22,13,46,15,66,57,48,19)
b=c(1,2,3,4,5,6,7,8,9)
c=c(12,222,243,464,659,936,NA, NA, NA)
d=c(45,765,178,46,44,670,NA, NA, NA)
e=c(1,765,748,4,4,70,NA, NA, NA)
df = cbind(a,b,c,d,e)

df

       a b   c   d   e
 [1,] 12 1  12  45   1
 [2,] 22 2 222 765 765
 [3,] 13 3 243 178 748
 [4,] 46 4 464  46   4
 [5,] 15 5 659  44   4
 [6,] 66 6 936 670  70
 [7,] 57 7  NA  NA  NA
 [8,] 48 8  NA  NA  NA
 [9,] 19 9  NA  NA  NA

The missing values are predominantly shares varying between 0 and 1, only one of them is a numerical column. Also I should that the sum of observed shares always adds up to 1, i.e. they are linearly dependent. Columns with no missing variables include numerical and categorical variables. I use all of these variables for the regression model, including the columns with missing values. I impute all columns with Predictive Mean Matching method.

I know that having highly associated missingness delays convergence, i.e. one needs to perform more iterations to reach convergence. But my question is, is it possible to reach convergence if they are 100% correlated, theoretically and in practice?

To test this I checked convergence plots. Instead of 16 missing columns, I took only 2 columns, for the sake of simplicity. I imputed them three times with 10000 iterations with the following code:

mice(df_PMM_imp, m = 3, maxit = 10000, predictorMatrix = pred_max, print = FALSE)

pred_max here is used only to exclude some irrelevant variables from the regression phase of Predictive Mean Matching.

The result I got is the following:

The averages observed values of the food and education variables are respectively 0,179154 and 0.005191 while the percentage of imputed zeros are respectively 0% and 87%.

For the imputation presented by the red line the averages of imputations for food and education variables are respectively 0,177593 and 0.006761 while the percentage of imputed zeros are respectively 0% and 40%.

For the imputation presented by the green line the averages of imputations for food and education variables are respectively 0,177060 and 0.007298 while the percentage of imputed zeros are respectively 0% and 39%.

Is it normal that the means and sd graphs of the red and other lines are so obviously separated? How would you evaluate these graphs and values? Do they imply that I need to have a much higher number of iterations? As if 10000 is not already enough... To me, judging by the graphs, there is something strange going on here. In all imputations, we see that the imputed averages are getting farther and farther away from the real value...

Any insight would be appreciated.

Thanks for your help in advance.

Answer 1

My guess is that your problem comes from the linear dependence among the set of "shares" variables. In any event your imputation method doesn't seem to maintain that logically necessary linear dependence, so you need to change the way you are doing the imputation even if that isn't leading to slow convergence.

The set of "shares" variables evidently represents the proportion of spending on different classes of items, with the sum of all those "shares" variables necessarily equalling 1. Predictive mean matching for each of those "shares" variables separately, the default in mice, will not respect the critical constraint that they add up to 1. Although I can't give a formal proof that this causes your slow convergence, it seems likely to be the source. And even if this isn't the source of your slow convergence, your imputed data wouldn't really make sense unless those shares all add up to 1 in each instance.

One solution would be to use settings and imputation methods other than the defaults in mice to deal with that constraint. Another, probably better solution (if you have information on total spending for each case with known "shares"), would be to work with the actual spending amounts in each category and then calculate the "shares" from those amounts.

It's possible to write special-purpose imputation methods for mice, providing ways to deal with constraints as explained on this page. The mice package itself provides a mice.impute.passive() function that performs calculations based on the imputed data. The first way I thought to proceed was to impute all but one of the "shares" values and then calculate the last "share" as 1 minus the sum of the other shares with mice.impute.passive(), but that could give a meaningless negative "share" in some cases.

One solution would be to impute all the "shares" together in each case as a categorical distribution. For each case with missing shares, find a set of nearest-neighbor cases as in predictive mean matching, randomly choose one nearest neighbor, and then use its entire set of "shares" as the imputed values.

If you have information on actual spending instead of on "shares" I would prefer imputing the actual spending values and then using mice.impute.passive() to calculate the corresponding "shares"--if you really want to use "shares" in your ultimate model. Modeling can be better and more interpretable if it's done on actual values rather than on ratios like your "shares" variables.