I have a dataset with missing values which I would like to impute by using Multiple Imputation by Chained Equations (MICE). The important characteristics of the dataset is that, for the columns containing missing values, a household either lacks all the values or has all of them. I give a toy dataset with the same feature below: (only that my data has 36 columns, of which 18 contains missing variables, and 6119 rows, of which 1529 contains missing variables)
a=c(12,22,13,46,15,66,57,48,19)
b=c(1,2,3,4,5,6,7,8,9)
c=c(12,222,243,464,659,936,NA, NA, NA)
d=c(45,765,178,46,44,670,NA, NA, NA)
e=c(1,765,748,4,4,70,NA, NA, NA)
df = cbind(a,b,c,d,e)
df
a b c d e
[1,] 12 1 12 45 1
[2,] 22 2 222 765 765
[3,] 13 3 243 178 748
[4,] 46 4 464 46 4
[5,] 15 5 659 44 4
[6,] 66 6 936 670 70
[7,] 57 7 NA NA NA
[8,] 48 8 NA NA NA
[9,] 19 9 NA NA NA
The missing values are predominantly shares varying between 0 and 1, only one of them is a numerical column. Also I should that the sum of observed shares always adds up to 1, i.e. they are linearly dependent. Columns with no missing variables include numerical and categorical variables. I use all of these variables for the regression model, including the columns with missing values. I impute all columns with Predictive Mean Matching method.
I know that having highly associated missingness delays convergence, i.e. one needs to perform more iterations to reach convergence. But my question is, is it possible to reach convergence if they are 100% correlated, theoretically and in practice?
To test this I checked convergence plots. Instead of 16 missing columns, I took only 2 columns, for the sake of simplicity. I imputed them three times with 10000 iterations with the following code:
mice(df_PMM_imp, m = 3, maxit = 10000, predictorMatrix = pred_max, print = FALSE)
pred_max here is used only to exclude some irrelevant variables from the regression phase of Predictive Mean Matching.
The result I got is the following:
The averages observed values of the food and education variables are respectively 0,179154 and 0.005191 while the percentage of imputed zeros are respectively 0% and 87%.
For the imputation presented by the red line the averages of imputations for food and education variables are respectively 0,177593 and 0.006761 while the percentage of imputed zeros are respectively 0% and 40%.
For the imputation presented by the green line the averages of imputations for food and education variables are respectively 0,177060 and 0.007298 while the percentage of imputed zeros are respectively 0% and 39%.
Is it normal that the means and sd graphs of the red and other lines are so obviously separated? How would you evaluate these graphs and values? Do they imply that I need to have a much higher number of iterations? As if 10000 is not already enough... To me, judging by the graphs, there is something strange going on here. In all imputations, we see that the imputed averages are getting farther and farther away from the real value...
Any insight would be appreciated.
Thanks for your help in advance.
mice
? For example, which specific imputation methods were used, what are the data types of missing values, how was the set of predictors for imputing within each column selected, are there columns of data that might be useful for imputation even if they are not included in your model after imputation, and so forth. Also, make sure to select a specific random seed value before you start the imputation, to allow for reproducible results from time to time. Do you have plots of convergence with iteration number? $\endgroup$