missing value patterns

Question

I am doing some data preparation with Python using Pandas and I am working with a dataset that has about 80 variables with missing values and I want to capture any patterns of missingness to cut down on the amount of missing value indicators I have but I am having trouble finding any good strategies for doing this. here is an example of what I am getting at:

MISS_1 MISS_2 MISS_3 MISS_4 MVP1 MVP2 MVP3 MISS_STR
     0      0      1      0    0    0    1     0010
     1      0      1      0    1    0    0     1010
     1      1      1      1    0    1    0     1111
     1      1      1      1    0    1    0     1110
     1      0      1      0    1    0    0     1010
     0      0      1      0    0    0    1     0010
     0      0      1      0    0    0    1     0010
     0      0      1      0    0    0    1     0010
     1      0      1      0    1    0    0     1010

One thing I tried was to create a string variable that concatenated all the missing value indicator variables like the 'MISS_STR', unfortunately the number of unique values in this variable was around 2000. Also, I know I am missing patterns that may be useful because if a pattern exists between Variable_1 and Variable_2 and Variable_3 and Variable_4 are missing completely at random, then concatenating them as strings will not capture the pattern between Variable_1 and Variable_2. Is there a better way of doing this? A good clustering algorithm perhaps? Whatever I do, I need to be able to do the same thing to my scoring data.

Answer 1

If I understand you correctly, you have several binary features and you would like to know if they have any correlation. Exploratory work is always hard but I would think something like Principal Component Analysis would give you a good idea of the relative correlations between your features.

Also, you would have much better luck if you removed the coding part from the question. While python is great, questions belong in Stack Overflow.