I have a training data set and I was able to find some interesting patterns in the missing values, and I used binary variables in order to represent the missingness. I am going to train a model, say a random forest, but I am unsure as to how to utilize my missing value indicators. Do I need to create the same variables (obviously different patterns) in the test set and then run the model? I assume that this is what I need to do, but I was not sure if there was a way that I could do this automatically.

This is not just a programming issue. I am sort of confused as to how to utilize the missing value pattern. Do I cluster observations based on the pattern? Do I explicitly utilize the missing value indicators?

  • $\begingroup$ More suitable for Stack Overflow? $\endgroup$ – Gavin M. Jones Jul 22 '15 at 1:43
  • $\begingroup$ The easy hack for randomForest: For numeric features replace NA or similar with the number C being a far outlier from any other values of that feature. For catagorical, assign missing a new label/class. Remember when utilizing such missing values, you get depended on these values to reappear in the same manor in future predictions. You could also use a permute function such as RFpermute in R to replace missing values, and perhaps add a new column describing missing values. $\endgroup$ – Soren Havelund Welling Jul 23 '15 at 7:42
  • $\begingroup$ Related: stats.stackexchange.com/questions/98953/… $\endgroup$ – Sycorax Dec 5 '19 at 22:10

I think that I have it:

Let's say that you have some data with variables Y, X1, X2, and X3.

You then create three missing value indicators: M1, M2, and M3.

You train and validate you model, and, two weeks later, your boss gives you data that he wants you to predict. This (test) data has Y, X1, X2, and X3, but it obviously does not have M1, M2, or M3: that is, you must first create these variables. After creating the three variables you can then use your model in order to score the data that was given to you by your boss (i.e. the test data set). Any thoughts would be appreciated!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.