7
$\begingroup$

I'm working on a binary classification problem, with about 1000 binary features in total. The problem is that for each datapoint, I only know the values of a small subset of the features (around 10-50), and the features in this subset are pretty much random.

What's a good way to deal with the problem of the missing features? Is there a particular classification algorithm that handles missing features well? (Naive Bayes should work, but is there anything else?) I'm guessing I don't want to do some kind of variable imputation, since I have so many missing features.

$\endgroup$
4
  • 1
    $\begingroup$ Some questions: How many datapoints do you have (I smell a prototypical overfitting problem) ? What do you mean when you say "the features in this subset are pretty much random" ? $\endgroup$ – mlwida Mar 8 '11 at 7:53
  • $\begingroup$ I have about 40,000 datapoints, so each feature appears in about 1000 rows. (Seems large enough not to be overfitting?) I meant that there aren't groups of features that often appear together (whether feature X is missing or not is independent of what other features are missing or not). $\endgroup$ – raegtin Mar 9 '11 at 0:34
  • $\begingroup$ Have you tried treating the value "missing" as just another label? $\endgroup$ – charles.y.zheng Mar 9 '11 at 2:10
  • 1
    $\begingroup$ Are the features missing truly at random? What is the mechanism of missingness? $\endgroup$ – whuber Mar 9 '11 at 17:37
3
$\begingroup$

Assuming data are considered missing completely at random (cf. @whuber's comment), using an ensemble learning technique as described in the following paper might be interesting to try:

Polikar, R. et al. (2010). Learn++.MF: A random subspace approach for the missing feature problem. Pattern Recognition, 43(11), 3817-3832.

The general idea is to train multiple classifiers on a subset of the variables that compose your dataset (like in Random Forests), but to use only the classifiers trained with the non-missing features for building the classification rule. Be sure to check what the authors call the "distributed redundancy" assumption (p. 3 in the preprint linked above), that is there must be some equally balanced redundancy in your features set.

$\endgroup$
1
$\begingroup$

If the features in the subset are random you can still impute values. However, if you have that much missing data, I would think twice about whether or not you really have enough valid data to do any kind of analysis.

The multiple imputation FAQ page ---->

http://www.stat.psu.edu/~jls/mifaq.html

$\endgroup$
3
  • 1
    $\begingroup$ The paper I cited in my response discussed the issue of imputation. In short, the training set should be dense enough and there's still the issue of bias that increases with the no. missing features. $\endgroup$ – chl Mar 10 '11 at 22:02
  • $\begingroup$ @chl Thanks - Did not know about that one. $\endgroup$ – Ralph Winters Mar 11 '11 at 1:05
  • $\begingroup$ Also, the possibility of imputing data depends on the way missing patterns arose (as pointed by @whuber, indirectly). $\endgroup$ – chl Mar 11 '11 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.