2
$\begingroup$

I have a huge dataset for a binary classification problem (about 1.5 million rows), and the feature space is quite large (145 dimension).

Some of these features are factors (YES, NO), but there is missing data. So my question is:

1 - Should i drop the missing data and lose information (the response matrix is quite sparse)

or

2 - Model these factors as 3 levels? ie, instead of (YES, NO) they become (YES, NO, X) where X is for missing data.

The problem with option 2 is that there is a lot of rows with missing information, so i would lose a lot of samples.

EDIT: Actually, the missing values happen when an specific condition happens, so they are informative as they are not 'missing' in the normal way. Does this reinforce the approach 2 above?

Thanks for any insight!

$\endgroup$
1
$\begingroup$

Option 1 is an alternative that must be considered, but there are other approaches, and combinations of approaches. Each column which possesses missing values must be treated individually.

The decision of how do deal with each column will depend on many factors: the meaning of the column, proportion of missing values, nature of missing values (if it's a categorical variable, a missing value can be even very informative to predict the response variable), etc. There is no "default" treatment. We need specific information to give specific advise.

You should deal with it as systematically as possible:

  1. List all columns which have missing values.
  2. Determine the proportion of missing values in each column.
  3. Choose standard candidate approaches for each column (list-wise deletion, mean imputation, regression imputation, etc.).
  4. Evaluate the best approaches (you could for example train your classifier with two different approaches and evaluate them in a validation set).

There are lots of advanced approaches. Everything said above apply to them anyway. Googling "missing data" will give you many more insights.

[Edit: comment about "option 2" removed, because the original question was modified and the comment is not applicable anymore]

$\endgroup$
  • $\begingroup$ Thanks, see my edit regarding your comment: 'missing value can be even very informative to predict the response variable'. $\endgroup$ – Fernando Oct 15 '14 at 20:51
  • $\begingroup$ Ok, now your approach 2 sounds better. You definitely should keep this information if it is informative. $\endgroup$ – Saul Berardo Oct 15 '14 at 21:02
0
$\begingroup$

It is possible to impute predictor variables, see e.g. http://www.epb.uni-hamburg.de/files/DiscussionPaper_1.pdf although it is necessary to assure they are missing at random and this approach should not be used if there are many missing predictors/row.

Discussion of several methods for imputation of binary data https://www.cs.uic.edu/pub/Isaim2012/WebPreferences/ISAIM2012_Boolean_Anthony_etal.pdf example in r Simultaneous imputation of multiple binary variables in R

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.