Continuous + categorical + Unavailable data

Question

I'm having some troubles with a classification task, and maybe the community could give me some advice. Here's my problem.

First, I had some continuous features and I had to say if the system was in the class 1, class 2 or class 3. This is a standard classification task, no big deal, the classifier could be a GMM or SVM etc., and it worked fine. The feature matrix looked like this:

\begin{array} {|l|rrrrrrrr|} \hline \textbf{Time}& T1 & T2 & T3 & T4 &T5 &T6 &T7 & ...\\ \hline \hline \textbf{Feat2}&0.2 &1 &0.15 &1.2 &10 &102 &120 &... \\ \hline \textbf{Feat2} &0.1 &0.11 &0.1 &0.2 &0.2 &0.1 &0.5 &...\\ \hline \textbf{...}& ...& ... &... &... &... &... &... &...\\ \hline \textbf{Label} & 0 &0 &1 &1 &1 &2 &2 & ...\\ \hline \end{array}

Now, I can have access to new data that I know could help. However, the data are categorical {1, 2, 3} but more importantly, sometimes they are not available. So my feature matrix looks now like this:

\begin{array} {|l|rrrrrrrr|} \hline \textbf{Time} & T1 & T2 & T3 & T4 &T5 &T6 &T7 & ...\\ \hline \hline \textbf{Feat1} &0.2 &1 &0.15 &1.2 &10 &102 &120 &... \\ \hline \textbf{Feat2} &0.1 &0.11 &0.1 &0.2 &0.2 &0.1 &0.5 &...\\ \hline \textbf{...} & ...& ... &... &... &... &... &... &...\\ \hline \textbf{Y1} &NA &NA &1 &1 &1 &NA &2 &...\\ \hline \textbf{Y2} &NA &0 &NA &NA &1 &2 &NA &...\\ \hline \textbf{Y3} &NA &NA &2 &0 &NA &NA &NA &...\\ \hline \textbf{...}& ...& ... &... &... &... &... &... &...\\ \hline \textbf{Label} & 0 &0 &1 &1 &1 &2 &2 & ...\\ \hline \end{array}

NA = Not Available.

Some data are irrelevant, but I know that some could be useful. In this example, I know that $Y1$ is valuable because when it is available, it matches the label. So my question is: how can I handle these data?

I know that categorical data can be converted into numerical data and then be used as the rest of the continuous features but how to manage the fact that they are sometimes unavailable? I tried to convert the "NA" into, let say -1, and then feed the classifier with the now complete data. For instance, $Y1$ becomes:

\begin{array} {|l|rrrrrrr|} \hline \textbf{Time} & T1 & T2 & T3 & T4 &T5 &T6 &T7 & ...\\ \hline \textbf{Y1} &-1 &-1 &1 &1 &1 &-1 &2 &...\\ \hline \end{array}

But it doesn't work, and the classification accuracy drops (which is not surprising since I feed the classifier with data that are irrelevant most of the time, i.e. for different classes they give the same output: -1).

Ideally, I would like that the classifier uses the data in a more efficient way. The classifier should use the common features but also take into account the availability of the new features Y, like "if Y is available, I can rely on it, otherwise, I use the standard feature".

How should I treat these data? Should I change the classifier? With the previous statement, it looks like I should add a Decision Tree or something, but at first I didn't want to add another classification step.

Has anyone got a thought on that? :)

Note: It also reminds me the "missing data problem" but I feel it's not the same case.

Answer 1

A common way to fill in the missing data would be to predict it, based on other objects that do have that data.

Depending on your datatypes, this could be a simple regression.

Treating all NA as the same numerical value seems doomed to fail, as you've seen.

Answer 2

You are dealing with a clasification problem with mixed variable types (continuous and categorical) along with missing data.

Now in this case, rather than SVM or GMM, a decision or a random forest classifier is more appropriate. These classifiers will work even in the presence of missing data.

And don't replace the missing data with -1 because the technique considers that it is its actual value. Replace it with a value so that the algorithm considers it as a missing value. For example, replace these values as NaN in Matlab or ? in Weka, according to the programming platform.