My question is directed to techniques to deal with incomplete data during the classifier/model training/fitting.

For instance, in a dataset w/ a few hundred rows, each row having let's say five dimensions and a class label as the last item, most data points will look like this:

[0.74, 0.39, 0.14, 0.33, 0.34, 0]

A few might look something like this:

[0.21, 0.68, ?, 0.82, 0.58, 1]

So it's those types of data points that are the focus of this Question.

My initial reason for asking this question was a problem directly in front of me; however, before posting my Question, i thought it might be more useful if i re-phrased it so the answers would be useful to a larger portion of the Community.

As a simple heuristic, let's divide these data-handling techniques based on when during the processing flow they are employed--before input to the classifier or during (i.e., the technique is inside the classifier).

The best example i can think of for the latter is the clever 'three-way branching' technique used in Decision Trees.

No doubt, the former category is far larger. The techniques i am aware of all fall into one of the groups below.

While recently reviewing my personal notes on "missing data handling" i noticed that i had quite an impressive list of techniques. I just maintain these notes for general peace of mind and in case a junior colleague asks me how to deal with missing data. In actual practice, i don't actually use any of them, except for the last one.

  1. Imputation: a broad rubric for a set of techniques which whose common denominator (i believe) is that the missing data is supplied directly by the same data set--substitution rather than estimation/prediction.

  2. Reconstruction: estimate the missing data points using an auto-associative network (just a neural network in which the sizes of the input and output layers are equal--in other words, the output has the same dimension as the input); the idea here is to train this network on complete data, then feed it incomplete patterns, and read the missing values from the output nodes.

  3. Bootstrapping: (no summary necessary i shouldn't think, given it's use elsewhere in statistical analysis).

  4. Denial: quietly remove the data points with missing/corrupt elements from your training set and pretend they never existed.

  • 2
    $\begingroup$ There's also "reduced-model" approach where you train a classifier for every pattern of missing values encountered during testing. IE, to make prediction for x where i'th attribute is missing, remove i'th attribute from all instances of training data and train on that. jmlr.csail.mit.edu/papers/v8/saar-tsechansky07a.html $\endgroup$ Commented Aug 10, 2010 at 1:27
  • $\begingroup$ I believe your definition of Imputation is incorrect in the modern context. Imputation now involves modeling the missing data based on other variables from the data set. The currently-favored Imputation method is Multiple Imputation, which generates multiple alternatives for each missing value (based on the model), processes each alternative completed data set, and then combines the answers reflecting the variability between the results. (In the "old days", people did things like substitute the value from a similar record, or the mean, etc, but not now.) $\endgroup$
    – Wayne
    Commented Feb 19, 2012 at 14:56
  • $\begingroup$ @Wayne would you be so kind to point me to some paper describing these modern techniques? Thanks $\endgroup$
    – Enzo
    Commented Apr 13, 2017 at 12:07
  • $\begingroup$ The R package mice has a nice introductory paper on JSS: jstatsoft.org/article/view/v045i03 . (You should find the introduction useful, even if you don't use R.) And the R package Amelia has a nice vignette that's included with the package. These two packages differ in their details, but both use multiple imputation. $\endgroup$
    – Wayne
    Commented Apr 13, 2017 at 12:17

3 Answers 3


I gave this answer to another question, but it might apply here too.

"There is a reasonably new area of research called Matrix Completion, that probably does what you want. A really nice introduction is given in this lecture by Emmanuel Candes"

Essentially, if your dataset has low rank (or approximately low rank) i.e. you have 100 rows, but the actual matrix has some small rank, say 10 (or only 10 large singular values), then you can use Matrix Completion to fill the missing data.


I might be a little unorthodox here, but what the heck. Please note: this line of thought comes from my own philosophy for classification, which is that I use it when my purpose is squarely on pure prediction -- not explanation, conceptual coherence, etc. Thus, what I'm saying here contradicts how I'd approach building a regression model.

Different classification approaches vary in their capability to handle missing data, and depending on some other factors^, I might just try #5: use a classifier that won't choke on those NAs. Part of the decision to go that route might also include thinking about how likely a similar proportion of NAs are to occur in the future data to which you'll be applying the model. If NAs for certain variables are going to be par for the course, then it might make sense to just roll with them (i.e., don't build a predictive model that assumes more informative data than what you'll actually have, or you'll be kidding yourself about how predictive it really is going to be). In fact, if I'm not convinced that NAs are missing at random, I'd be inclined to recode a new variable (or a new level if it's missing in a categorical variable) just to see if the missingness itself is predictive.

If I had a good reason to use a classifier that did not take missing data very well, then my approach would be #1 (multiple imputation), seeking to find a classification model that behaved similarly well across imputed data sets.

^Including: how much missingness you have in your predictors, whether there are systematic patterns (if there are, it would be worth taking a closer look and thinking through the implications for your analysis), and how much data you have to work with overall.


If you have a reasonable hunch about the data generating process that is responsible for the data in question then you could use bayesian ideas to estimate the missing data. Under the bayesian approach you would simply assume that the missing data are also random variables and construct the posterior for the missing data conditional on the observed data. The posterior means would then be used as a substitute for the missing data.

The use of bayesian models may qualify as imputation under a broad sense of the term but I thought of mentioning it as it did not appear on your list.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.