Supervised classifier for events with missing data

Question

I'm working on an experiment in which we try to sort events into three categories based on roughly 20 inputs using a feed-forward neural network (trained via backpropagation). Unfortunately, many variables which offer good discrimination become meaningless in some situations. So far, the procedure has been to set meaningless variables to some placeholder value which lies outside the normal range for that variable.

But that's roughly half the inputs in something like half the events. The classification isn't as strong as we'd like, so the options I've come up with are:

1. Split the events based on whether the unreliable variables are present, train several neural nets on the sub-samples. The classified samples would then be merged.

2. Switch to some other classifier (boosted decision trees are very popular in my field) which handles missing information more naturally.

Is one of these approaches obviously superior? Is there some other obvious option I'm missing?

Answer 1

As requested, I'll elaborate on my comment, although I don't have experience using it. I work with neural networks for regression problems and often construct new features, but I don't have to deal with missing data so I'm not sure whether this will work.

Let's suppose the features of your data look like

$(0,1)$

$(0.5,0.5)$

$(*,0.2)$

$(*,0.7)$

$(0.8,*)$

where $*$ means the value is missing or unreliable. Rather than replacing $*$ with a very large or very small value outside the typical range, or breaking up your data so that you have a separate net for each subset of the data which might be missing, I suggest making two features for each input which might be missing. If the value is present, the two features become $(0,value)$. If the value is missing, then the features become $(1,random)$ where you randomly sample value from the range. So, the above data set would become

$(0,0,0,1)$

$(0,0.5,0,0.5)$

$(1,rand,0,0.2)$

$(1,rand,0,0.7)$

$(0,0.8,1,rand)$

Each point with a random coordinate can be cloned to give several inputs which you can train with a lower learning rate.

$(0,0,0,1), \alpha = \alpha_0 $

$(0,0.5,0,0.5), \alpha = \alpha_0$

$(1,0,0,0.2), \alpha = \alpha_0/3$

$(1,0.5,0,0.2), \alpha = \alpha_0/3$

$(1,0.8,0,0.2), \alpha = \alpha_0/3$

$(1,0,0,0.7), \alpha = \alpha_0/3$

$(1,0.5,0,0.7), \alpha = \alpha_0/3$

$(1,0.8,0,0.7), \alpha = \alpha_0/3$

$(0,0.8,1,1), \alpha = \alpha_0/4$

$(0,0.8,1,0.5), \alpha = \alpha_0/4$

$(0,0.8,1,0.2), \alpha = \alpha_0/4$

$(0,0.8,1,0.7), \alpha = \alpha_0/4$

One idea is that this should encourage the neural network to learn that when the indicator that the value is missing is $1$, then the value doesn't matter. You can test whether this is true for the neural network, and perhaps use a regularizer which encourages this.

Answer 2

Doug Zare's suggestion is a little like a missing data technique called multiple imputation. The data point gets repeated many times with plausible values for the missing variable being input. I think that would allow the classifier to gain the information from the correct coordinate and in a way learn the uncertainty from the missing one. The tree classifiers can handle missing data with surrogate variables (i.e. using a replacement variable that is known to be highly correlated with the one that is missing).