I'm working on an experiment in which we try to sort events into three categories based on roughly 20 inputs using a feed-forward neural network (trained via backpropagation). Unfortunately, many variables which offer good discrimination become meaningless in some situations. So far, the procedure has been to set meaningless variables to some placeholder value which lies outside the normal range for that variable.

But that's roughly half the inputs in something like half the events. The classification isn't as strong as we'd like, so the options I've come up with are:

  1. Split the events based on whether the unreliable variables are present, train several neural nets on the sub-samples. The classified samples would then be merged.

  2. Switch to some other classifier (boosted decision trees are very popular in my field) which handles missing information more naturally.

Is one of these approaches obviously superior? Is there some other obvious option I'm missing?

  • $\begingroup$ I would add a feature for each value which might be missing indicating whether that value is missing, and clone the training point with several values for the missing data drawn from the normal ranges (using a lower learning rate to avoid overemphasizing that point). $\endgroup$ – Douglas Zare Aug 24 '12 at 8:47
  • $\begingroup$ @DouglasZare, can you elaborate more on that? (maybe an example will be good). I'm really interested in this question, but I'm not sure I really got you suggestion. $\endgroup$ – Néstor Aug 24 '12 at 17:36

As requested, I'll elaborate on my comment, although I don't have experience using it. I work with neural networks for regression problems and often construct new features, but I don't have to deal with missing data so I'm not sure whether this will work.

Let's suppose the features of your data look like






where $*$ means the value is missing or unreliable. Rather than replacing $*$ with a very large or very small value outside the typical range, or breaking up your data so that you have a separate net for each subset of the data which might be missing, I suggest making two features for each input which might be missing. If the value is present, the two features become $(0,value)$. If the value is missing, then the features become $(1,random)$ where you randomly sample value from the range. So, the above data set would become






Each point with a random coordinate can be cloned to give several inputs which you can train with a lower learning rate.

$(0,0,0,1), \alpha = \alpha_0 $

$(0,0.5,0,0.5), \alpha = \alpha_0$

$(1,0,0,0.2), \alpha = \alpha_0/3$

$(1,0.5,0,0.2), \alpha = \alpha_0/3$

$(1,0.8,0,0.2), \alpha = \alpha_0/3$

$(1,0,0,0.7), \alpha = \alpha_0/3$

$(1,0.5,0,0.7), \alpha = \alpha_0/3$

$(1,0.8,0,0.7), \alpha = \alpha_0/3$

$(0,0.8,1,1), \alpha = \alpha_0/4$

$(0,0.8,1,0.5), \alpha = \alpha_0/4$

$(0,0.8,1,0.2), \alpha = \alpha_0/4$

$(0,0.8,1,0.7), \alpha = \alpha_0/4$

One idea is that this should encourage the neural network to learn that when the indicator that the value is missing is $1$, then the value doesn't matter. You can test whether this is true for the neural network, and perhaps use a regularizer which encourages this.

  • $\begingroup$ this is an interesting idea. I'm not sure about using a flat random distribution, though: probably better to pick the value from a distribution that approximates the data, no? $\endgroup$ – Shep Aug 24 '12 at 19:40
  • $\begingroup$ Yes, that's what I did in the example. $\endgroup$ – Douglas Zare Aug 24 '12 at 20:32

Doug Zare's suggestion is a little like a missing data technique called multiple imputation. The data point gets repeated many times with plausible values for the missing variable being input. I think that would allow the classifier to gain the information from the correct coordinate and in a way learn the uncertainty from the missing one. The tree classifiers can handle missing data with surrogate variables (i.e. using a replacement variable that is known to be highly correlated with the one that is missing).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.