# Clustering a dataset with both discrete and continuous variables

I have a dataset X which has 10 dimensions, 4 of which are discrete values. In fact, those 4 discrete variables are ordinal, i.e. a higher value implies a higher/better semantic.

2 of these discrete variables are categorical in the sense that for each of these variables, the distance e.g. from 11 to 12 is not the same as the distance from 5 to 6. While a higher variable value implies a higher in reality, the scale is not necessarily linear (in fact, it is not really defined).

My question is:

• Is it a good idea to apply a common clustering algorithm (e.g. K-Means and then Gaussian Mixture (GMM)) to this dataset which contains both discrete and continuous variables?

If not:

• Should I remove the discrete variables and focus only on the continuous ones?
• Should I better discretize the continuous ones and use a clustering algorithm for discrete data?
• You need to find a good distance measure (often the most difficult task in clustering): if you can find a distance measure that correctly and accurately describes how similar (or not) your data items are, then you should not have any problems. – Andrew May 10 '12 at 10:59
• Speaking about those 2 categorical variables you in effect described them as ordinal. Now, what's about the rest 2 "ordinal" variables? How are they different from those? – ttnphns May 10 '12 at 14:12
• They are also discrete, but both of them have a meaningful distance function, i.e. they are interval-based (if I am not messing up the definition of interval-based). – ptikobj May 10 '12 at 14:43

So you've been told you need an appropriate distance measure. Here are some leads:

and, of course: Mahalanobis distance.

I've had to deal with this kind of problem in the past, and I think there could be 2 interesting approaches:

• Continuousification: transform symbolic attributes with a sequence of integers. There are several ways to do this, all of which described in this paper. You can try NBF, VDM and MDV algorithms.

• Discretization: transform continuous attributes into symbolic values. Again, many algorithms, and a good lecture on this would be this article. I believe the most commonly used method is Holte's 1R, but the best way to know for sure is to look at the ROC curves against algorithms like EWD, EFD, ID, LD or NDD.

Once you have all your features in the same space, it becomes an usual clustering problem.

Choosing between continuousification or discretization depends on your dataset and what your features look like, so it's a bit hard to say, but I advise you to read the articles I gave you on that topic.

K-means obviously doesn't make any sense, as it computes means (which are nonsensical). Same goes for GMM.

You might want to try distance-based clustering algorithms with appropriate distance functions, for example DBSCAN.

The main challenge is to find a distance function!

While you could put a different distance function into k-means, it will still compute the mean which probably doesn't make much sense (and probably messes with a distance function for discrete values).

Anyway, first focus on define what "similar" is. Then cluster using this definition of similar!

If you are comfortable working with a distance matrix of size num_of_samples x num_of_samples, you could use random forests, as well.

Click here for a reference paper titled Unsupervised learning with random forest predictors.

The idea is creating a synthetic dataset by shuffling values in the original dataset and training a classifier for separating both. During classification you will get an inter-sample distance matrix, on which you could test your favorite clustering algorithm.

Mixed approach to be adopted: 1) Use classification technique (C4.5 decision tree) to classify the data set into 2 classes. 2) Once it is done, leave categorical variables and proceed with continuous variables for clustering.

• I could not follow your suggestion. Which two classes, and how will that help? – KarthikS Jun 11 '15 at 10:07
• I think what Swapnil Soni needs to say is that once we use the classification technique to classify it into two class. We can then use the label of classification output as a binary variable. So instead of all the categorical variable you get an indicative binary variable and then your clustering algorithm can proceed with the data ( consisting of all continuous plus 1 binary variable). My interpretation can be wrong though. – Tusharshar Jun 23 '15 at 11:33
• perfectly fine! – Swapnil Soni Aug 17 '15 at 6:18