I am doing some data preparation with Python using Pandas and I am working with a dataset that has about 80 variables with missing values and I want to capture any patterns of missingness to cut down on the amount of missing value indicators I have but I am having trouble finding any good strategies for doing this. here is an example of what I am getting at:
MISS_1 MISS_2 MISS_3 MISS_4 MVP1 MVP2 MVP3 MISS_STR
0 0 1 0 0 0 1 0010
1 0 1 0 1 0 0 1010
1 1 1 1 0 1 0 1111
1 1 1 1 0 1 0 1110
1 0 1 0 1 0 0 1010
0 0 1 0 0 0 1 0010
0 0 1 0 0 0 1 0010
0 0 1 0 0 0 1 0010
1 0 1 0 1 0 0 1010
One thing I tried was to create a string variable that concatenated all the missing value indicator variables like the 'MISS_STR', unfortunately the number of unique values in this variable was around 2000. Also, I know I am missing patterns that may be useful because if a pattern exists between Variable_1 and Variable_2 and Variable_3 and Variable_4 are missing completely at random, then concatenating them as strings will not capture the pattern between Variable_1 and Variable_2. Is there a better way of doing this? A good clustering algorithm perhaps? Whatever I do, I need to be able to do the same thing to my scoring data.