7
$\begingroup$

I'm working on a binary classification problem, with about 1000 binary features in total. The problem is that for each datapoint, I only know the values of a small subset of the features (around 10-50), and the features in this subset are pretty much random.

What's a good way to deal with the problem of the missing features? Is there a particular classification algorithm that handles missing features well? (Naive Bayes should work, but is there anything else?) I'm guessing I don't want to do some kind of variable imputation, since I have so many missing features.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Some questions: How many datapoints do you have (I smell a prototypical overfitting problem) ? What do you mean when you say "the features in this subset are pretty much random" ? $\endgroup$
    – steffen
    Commented Mar 8, 2011 at 7:53
  • $\begingroup$ I have about 40,000 datapoints, so each feature appears in about 1000 rows. (Seems large enough not to be overfitting?) I meant that there aren't groups of features that often appear together (whether feature X is missing or not is independent of what other features are missing or not). $\endgroup$
    – raegtin
    Commented Mar 9, 2011 at 0:34
  • $\begingroup$ Have you tried treating the value "missing" as just another label? $\endgroup$
    – charles.y.zheng
    Commented Mar 9, 2011 at 2:10
  • 1
    $\begingroup$ Are the features missing truly at random? What is the mechanism of missingness? $\endgroup$
    – whuber ♦
    Commented Mar 9, 2011 at 17:37

2 Answers 2

Reset to default
3
$\begingroup$

Assuming data are considered missing completely at random (cf. @whuber's comment), using an ensemble learning technique as described in the following paper might be interesting to try:

Polikar, R. et al. (2010). Learn++.MF: A random subspace approach for the missing feature problem. Pattern Recognition, 43(11), 3817-3832.

The general idea is to train multiple classifiers on a subset of the variables that compose your dataset (like in Random Forests), but to use only the classifiers trained with the non-missing features for building the classification rule. Be sure to check what the authors call the "distributed redundancy" assumption (p. 3 in the preprint linked above), that is there must be some equally balanced redundancy in your features set.

Improve this answer
$\endgroup$
1
$\begingroup$

If the features in the subset are random you can still impute values. However, if you have that much missing data, I would think twice about whether or not you really have enough valid data to do any kind of analysis.

The multiple imputation FAQ page ---->

http://www.stat.psu.edu/~jls/mifaq.html

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ The paper I cited in my response discussed the issue of imputation. In short, the training set should be dense enough and there's still the issue of bias that increases with the no. missing features. $\endgroup$
    – chl
    Commented Mar 10, 2011 at 22:02
  • $\begingroup$ @chl Thanks - Did not know about that one. $\endgroup$
    – Ralph Winters
    Commented Mar 11, 2011 at 1:05
  • $\begingroup$ Also, the possibility of imputing data depends on the way missing patterns arose (as pointed by @whuber, indirectly). $\endgroup$
    – chl
    Commented Mar 11, 2011 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.