Techniques for Handling Incomplete/Missing Data

Question

My question is directed to techniques to deal with incomplete data during the classifier/model training/fitting.

For instance, in a dataset w/ a few hundred rows, each row having let's say five dimensions and a class label as the last item, most data points will look like this:

[0.74, 0.39, 0.14, 0.33, 0.34, 0]

A few might look something like this:

[0.21, 0.68, ?, 0.82, 0.58, 1]

So it's those types of data points that are the focus of this Question.

My initial reason for asking this question was a problem directly in front of me; however, before posting my Question, i thought it might be more useful if i re-phrased it so the answers would be useful to a larger portion of the Community.

As a simple heuristic, let's divide these data-handling techniques based on when during the processing flow they are employed--before input to the classifier or during (i.e., the technique is inside the classifier).

The best example i can think of for the latter is the clever 'three-way branching' technique used in Decision Trees.

No doubt, the former category is far larger. The techniques i am aware of all fall into one of the groups below.

While recently reviewing my personal notes on "missing data handling" i noticed that i had quite an impressive list of techniques. I just maintain these notes for general peace of mind and in case a junior colleague asks me how to deal with missing data. In actual practice, i don't actually use any of them, except for the last one.

1. Imputation: a broad rubric for a set of techniques which whose common denominator (i believe) is that the missing data is supplied directly by the same data set--substitution rather than estimation/prediction.

2. Reconstruction: estimate the missing data points using an auto-associative network (just a neural network in which the sizes of the input and output layers are equal--in other words, the output has the same dimension as the input); the idea here is to train this network on complete data, then feed it incomplete patterns, and read the missing values from the output nodes.

3. Bootstrapping: (no summary necessary i shouldn't think, given it's use elsewhere in statistical analysis).

4. Denial: quietly remove the data points with missing/corrupt elements from your training set and pretend they never existed.

Answer 1

I gave this answer to another question, but it might apply here too.

"There is a reasonably new area of research called Matrix Completion, that probably does what you want. A really nice introduction is given in this lecture by Emmanuel Candes"

Essentially, if your dataset has low rank (or approximately low rank) i.e. you have 100 rows, but the actual matrix has some small rank, say 10 (or only 10 large singular values), then you can use Matrix Completion to fill the missing data.

Answer 2

I might be a little unorthodox here, but what the heck. Please note: this line of thought comes from my own philosophy for classification, which is that I use it when my purpose is squarely on pure prediction -- not explanation, conceptual coherence, etc. Thus, what I'm saying here contradicts how I'd approach building a regression model.

Different classification approaches vary in their capability to handle missing data, and depending on some other factors^, I might just try #5: use a classifier that won't choke on those NAs. Part of the decision to go that route might also include thinking about how likely a similar proportion of NAs are to occur in the future data to which you'll be applying the model. If NAs for certain variables are going to be par for the course, then it might make sense to just roll with them (i.e., don't build a predictive model that assumes more informative data than what you'll actually have, or you'll be kidding yourself about how predictive it really is going to be). In fact, if I'm not convinced that NAs are missing at random, I'd be inclined to recode a new variable (or a new level if it's missing in a categorical variable) just to see if the missingness itself is predictive.

If I had a good reason to use a classifier that did not take missing data very well, then my approach would be #1 (multiple imputation), seeking to find a classification model that behaved similarly well across imputed data sets.

^Including: how much missingness you have in your predictors, whether there are systematic patterns (if there are, it would be worth taking a closer look and thinking through the implications for your analysis), and how much data you have to work with overall.

Answer 3

If you have a reasonable hunch about the data generating process that is responsible for the data in question then you could use bayesian ideas to estimate the missing data. Under the bayesian approach you would simply assume that the missing data are also random variables and construct the posterior for the missing data conditional on the observed data. The posterior means would then be used as a substitute for the missing data.

The use of bayesian models may qualify as imputation under a broad sense of the term but I thought of mentioning it as it did not appear on your list.