I would like to know how to handle missing data in predictive analysis:

In my case, missing information has been decided not to be omitted, however, certain predictive models such as logistic regression, random forest, couldn't handle missing data. So for this reason I have decided to do some data imputation before modelling.

Like all predictive analysis I have a training set and a test set. My confusion is that when I impute the training data, how can I then be able to handle the test data with possible missing information?

  • $\begingroup$ I don't understand what you are saying. It is confusing. Are you using an imputation method or not? Please give this some clarification. $\endgroup$ – Michael Chernick Mar 28 '17 at 14:38
  • $\begingroup$ @MichaelChernick Thanks for the reply. The imputation method does not matter I think. My question is: how do we handle the missing information simultaneously in a training and testing set? $\endgroup$ – user95902 Mar 28 '17 at 14:46
  • 1
    $\begingroup$ Tree-based model can handle the missing value! If you choose impute missing value, then impute the training data and testing data separately. $\endgroup$ – wolfe Mar 28 '17 at 14:56
  • $\begingroup$ @7-th Thank you! That's what I want to ask: is the model from an imputed training set still valid for a separately imputed test set? $\endgroup$ – user95902 Mar 28 '17 at 14:59
  • 1
    $\begingroup$ Yes, it's valid, if the imputed method is the same. Tree-based model is not to imputed the missing value, but using the proportion of missing value to calculate the feature's information gain. $\endgroup$ – wolfe Mar 28 '17 at 15:05

From my experience, there are two options:

  1. Discard instances where there are missing values (do this for both training and test set)
  2. Impute the missing values with the mean value (taking the mean value of the consolidated data - both training and test data)

There are more methods which you can find in this paper

Also, for both options above, you may consider discarding the data fields where the frequency of missing values is high in your model.

  • $\begingroup$ Imputation using the mean has long been supplanted by other more nimble methods. I like your last point though. $\endgroup$ – rolando2 Mar 28 '17 at 15:23
  • $\begingroup$ @rolando2: Can you explain or list the "more nimble methods"? $\endgroup$ – stackoverflowuser2010 Aug 1 '17 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.