I'm having some troubles with a classification task, and maybe the community could give me some advice. Here's my problem.

First, I had some continuous features and I had to say if the system was in the class 1, class 2 or class 3. This is a standard classification task, no big deal, the classifier could be a GMM or SVM etc., and it worked fine. The feature matrix looked like this:

\begin{array} {|l|rrrrrrrr|} \hline \textbf{Time}& T1 & T2 & T3 & T4 &T5 &T6 &T7 & ...\\ \hline \hline \textbf{Feat2}&0.2 &1 &0.15 &1.2 &10 &102 &120 &... \\ \hline \textbf{Feat2} &0.1 &0.11 &0.1 &0.2 &0.2 &0.1 &0.5 &...\\ \hline \textbf{...}& ...& ... &... &... &... &... &... &...\\ \hline \textbf{Label} & 0 &0 &1 &1 &1 &2 &2 & ...\\ \hline \end{array}

Now, I can have access to new data that I know could help. However, the data are categorical {1, 2, 3} but more importantly, sometimes they are not available. So my feature matrix looks now like this:

\begin{array} {|l|rrrrrrrr|} \hline \textbf{Time} & T1 & T2 & T3 & T4 &T5 &T6 &T7 & ...\\ \hline \hline \textbf{Feat1} &0.2 &1 &0.15 &1.2 &10 &102 &120 &... \\ \hline \textbf{Feat2} &0.1 &0.11 &0.1 &0.2 &0.2 &0.1 &0.5 &...\\ \hline \textbf{...} & ...& ... &... &... &... &... &... &...\\ \hline \textbf{Y1} &NA &NA &1 &1 &1 &NA &2 &...\\ \hline \textbf{Y2} &NA &0 &NA &NA &1 &2 &NA &...\\ \hline \textbf{Y3} &NA &NA &2 &0 &NA &NA &NA &...\\ \hline \textbf{...}& ...& ... &... &... &... &... &... &...\\ \hline \textbf{Label} & 0 &0 &1 &1 &1 &2 &2 & ...\\ \hline \end{array}

NA = Not Available.

Some data are irrelevant, but I know that some could be useful. In this example, I know that $Y1$ is valuable because when it is available, it matches the label. So my question is: how can I handle these data?

I know that categorical data can be converted into numerical data and then be used as the rest of the continuous features but how to manage the fact that they are sometimes unavailable? I tried to convert the "NA" into, let say -1, and then feed the classifier with the now complete data. For instance, $Y1$ becomes:

\begin{array} {|l|rrrrrrr|} \hline \textbf{Time} & T1 & T2 & T3 & T4 &T5 &T6 &T7 & ...\\ \hline \textbf{Y1} &-1 &-1 &1 &1 &1 &-1 &2 &...\\ \hline \end{array}

But it doesn't work, and the classification accuracy drops (which is not surprising since I feed the classifier with data that are irrelevant most of the time, i.e. for different classes they give the same output: -1).

Ideally, I would like that the classifier uses the data in a more efficient way. The classifier should use the common features but also take into account the availability of the new features Y, like "if Y is available, I can rely on it, otherwise, I use the standard feature".

How should I treat these data? Should I change the classifier? With the previous statement, it looks like I should add a Decision Tree or something, but at first I didn't want to add another classification step.

Has anyone got a thought on that? :)

Note: It also reminds me the "missing data problem" but I feel it's not the same case.


A common way to fill in the missing data would be to predict it, based on other objects that do have that data.

Depending on your datatypes, this could be a simple regression.

Treating all NA as the same numerical value seems doomed to fail, as you've seen.


You are dealing with a clasification problem with mixed variable types (continuous and categorical) along with missing data.

Now in this case, rather than SVM or GMM, a decision or a random forest classifier is more appropriate. These classifiers will work even in the presence of missing data.

And don't replace the missing data with -1 because the technique considers that it is its actual value. Replace it with a value so that the algorithm considers it as a missing value. For example, replace these values as NaN in Matlab or ? in Weka, according to the programming platform.

  • $\begingroup$ Thanks you for the answer, I didn't know the NaN trick for the decision tree. For now, I just tried to combine the outputs of the SVM model (which uses the continous features) and the categorical features using a Decision Tree. The accuracy is now better than the SVM alone so at least, it proves that the categorical features i'd like to use can be useful. The next step would be to combine both the categorical and continuous features within the same classifier : Decision Tree or Random Forest, like you mentioned. I'm not very familiar with RF so I have to learn this algorithm first. Thanks! $\endgroup$
    – JoKa
    Oct 30 '15 at 15:32
  • $\begingroup$ It will be good to learn RF. All the best!! $\endgroup$
    – prashanth
    Nov 2 '15 at 5:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.