2
$\begingroup$

I would like to know how to handle missing data in predictive analysis:

In my case, missing information has been decided not to be omitted, however, certain predictive models such as logistic regression, random forest, couldn't handle missing data. So for this reason I have decided to do some data imputation before modelling.

Like all predictive analysis I have a training set and a test set. My confusion is that when I impute the training data, how can I then be able to handle the test data with possible missing information?

Improve this question
$\endgroup$
9
  • $\begingroup$ I don't understand what you are saying. It is confusing. Are you using an imputation method or not? Please give this some clarification. $\endgroup$
    – Michael R. Chernick
    Commented Mar 28, 2017 at 14:38
  • $\begingroup$ @MichaelChernick Thanks for the reply. The imputation method does not matter I think. My question is: how do we handle the missing information simultaneously in a training and testing set? $\endgroup$
    – user95902
    Commented Mar 28, 2017 at 14:46
  • 1
    $\begingroup$ Tree-based model can handle the missing value! If you choose impute missing value, then impute the training data and testing data separately. $\endgroup$
    – wolfe
    Commented Mar 28, 2017 at 14:56
  • $\begingroup$ @7-th Thank you! That's what I want to ask: is the model from an imputed training set still valid for a separately imputed test set? $\endgroup$
    – user95902
    Commented Mar 28, 2017 at 14:59
  • 1
    $\begingroup$ Yes, it's valid, if the imputed method is the same. Tree-based model is not to imputed the missing value, but using the proportion of missing value to calculate the feature's information gain. $\endgroup$
    – wolfe
    Commented Mar 28, 2017 at 15:05

1 Answer 1

Reset to default
-1
$\begingroup$

From my experience, there are two options:

  1. Discard instances where there are missing values (do this for both training and test set)
  2. Impute the missing values with the mean value (taking the mean value of the consolidated data - both training and test data)

There are more methods which you can find in this paper

Also, for both options above, you may consider discarding the data fields where the frequency of missing values is high in your model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Imputation using the mean has long been supplanted by other more nimble methods. I like your last point though. $\endgroup$
    – rolando2
    Commented Mar 28, 2017 at 15:23
  • $\begingroup$ @rolando2: Can you explain or list the "more nimble methods"? $\endgroup$
    – stackoverflowuser2010
    Commented Aug 1, 2017 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.