Can we use clustering output as predictor variable for classification?

  1. I have a set of data and I do clustering analysis on it, it divides the data into different clusters.
  2. Can I use this cluster information i.e. cluster1, cluster2, cluster3, as one of the input variable i.e. predictor variable for my decision tree algorithm is it statistically OK? I found great increase in the model accuracy when I do this
  • $\begingroup$ yes you can do that, try it test it. in machine learning there is no need to justify, you can cross validate and test you model, if adding cluster variable means improved accuracy, use it by all means. $\endgroup$ – forecaster Apr 9 '14 at 0:17

You will face the problem how to assign new instances to their clusters.

The more common combination is to run cluster analysis to check if any class consists maybe of multiple clusters. Then use this information to train multiple classifiers for such classes (i.e. Class1A, Class1B, Class1C), and in the end strip the cluster information from the output (i.e. Class1A -> Class1).

If you are running cluster analysis first, then split your data, you have a data snooping bias problem. Don't do this. Running clusterig on the train and test set independently will usually not work, as clusters will often be very different.

In some rare cases (e.g. k-means) you can of course assign new instances to the nearest mean, with the means optimized on the training set only. But this is in fact no longer k-means, but NN-classification to a simplified data set (consisting of the means obtained by k-means). This approach is however only possible for the particular case of k-means with squared Euclidean distance.

  • $\begingroup$ Should this be a problem when i am dealing with only categorical variables (both predictor and predicted) and going to use decision tree. As i just want to know what combinations of predictor variable are giving me classes $\endgroup$ – Malu Mar 26 '14 at 6:48
  • $\begingroup$ Don't bother to use clustering at all then. Most clustering methods are designed for continuous values. When you want to analyze your categorial data in a classification context, why not analyze your decision tree or random forest? It's actually a best practise for analyzing variable importance. $\endgroup$ – Has QUIT--Anony-Mousse Mar 26 '14 at 9:55
  • $\begingroup$ Thanks for that help about data snooping bias. Although I may not be using clustering in present case. I could not understand what you mean to say in first para i.e. More common combination... Could you please elaborate little on that i.e. should this be done on entire data (i.e. training, test, and unkwon) or training set only? $\endgroup$ – Malu Mar 27 '14 at 5:20
  • $\begingroup$ This is independent of your validation process. If a class consists of multiple clusters, you may (or may not) get better result if you use separate detectors for each cluster. Say class A is 0-10 and 20-30, whereas class B is 10-20. Then results will usually be better when you first split class A into classes A1 and A2. $\endgroup$ – Has QUIT--Anony-Mousse Mar 29 '14 at 10:30

As @Peter Flom already stated, Yes, you can do this; however, keep in mind that you are sacrificing interpretation for prediction in this case (ie. what do changes in these clusters truly mean?). This may or may not be important to the yourself and/or the end user of your model.

If prediction (model accuracy, in your case) is the only thing that is important to you in this situation then by all means use the clustered variables as inputs in your model!

  • $\begingroup$ Thanks Matt for that insight. It will be a great help if yo could tell me. 1. Should i do clustering on whole data at once and then partition it into 1. training 2. Testing and 3. Unknowns or to be predicted data. Bcz if i first partition the data and then run cluster analysis then my clusters may not match within the datasets $\endgroup$ – Malu Mar 21 '14 at 11:49
  • $\begingroup$ Your thought process is correct! How large is your sample? How many of these clusters and other variables are going to be allowed to enter the model? I will further assist with this information. $\endgroup$ – Matt Reichenbach Mar 21 '14 at 13:09
  • $\begingroup$ I have about 2.3 million records, out which about 88k are known and rest are unknown (that i want to predict) i will be dividing into 10 clusters, and have 8 predictor variables $\endgroup$ – Malu Mar 26 '14 at 6:35
  • $\begingroup$ Unknown? As in the dependent variable? Is the dependent variable binary, categorical, or continuous? If it is binary or categorical, how many of the 88k records fall into the smallest group? $\endgroup$ – Matt Reichenbach Mar 26 '14 at 11:33
  • $\begingroup$ Sorry but my data has 880k known records and not 88k. Unknown means this are the records where dependent variable is to be predicted (means it is not part of 880k set). My data is categorical has 3 levels A, B, C with counts as 682866, 79120, 137470 respectively $\endgroup$ – Malu Mar 27 '14 at 5:10

Yes, you can do that. Why would you not be able to?

  • $\begingroup$ I thought we are creating data from the available data and putting it as an input. So i was just feeling that am i trying to create something artificial and being happy. could you please elaborate on the justification for the same $\endgroup$ – Malu Mar 21 '14 at 11:28
  • 1
    $\begingroup$ No justification is needed. $\endgroup$ – Peter Flom - Reinstate Monica Mar 21 '14 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.