Clustering a dataset with both discrete and continuous variables

Question

I have a dataset X which has 10 dimensions, 4 of which are discrete values. In fact, those 4 discrete variables are ordinal, i.e. a higher value implies a higher/better semantic.

2 of these discrete variables are categorical in the sense that for each of these variables, the distance e.g. from 11 to 12 is not the same as the distance from 5 to 6. While a higher variable value implies a higher in reality, the scale is not necessarily linear (in fact, it is not really defined).

My question is:

• Is it a good idea to apply a common clustering algorithm (e.g. K-Means and then Gaussian Mixture (GMM)) to this dataset which contains both discrete and continuous variables?

If not:

• Should I remove the discrete variables and focus only on the continuous ones?
• Should I better discretize the continuous ones and use a clustering algorithm for discrete data?

Answer 1

So you've been told you need an appropriate distance measure. Here are some leads:

• Clustering mixed data
• A generalized Mahalanobis distance for mixed data
• Estimating the Mahalanobis distance from mixed continuous and discrete data
• Generalization of the Mahalanobis distance in the mixed case
• Distance functions for categorical and mixed variables
• Informational distances and related statistics in mixed continuous and categorical variables

and, of course: Mahalanobis distance.

Answer 2

I've had to deal with this kind of problem in the past, and I think there could be 2 interesting approaches:

• Continuousification: transform symbolic attributes with a sequence of integers. There are several ways to do this, all of which described in this paper. You can try NBF, VDM and MDV algorithms.

• Discretization: transform continuous attributes into symbolic values. Again, many algorithms, and a good lecture on this would be this article. I believe the most commonly used method is Holte's 1R, but the best way to know for sure is to look at the ROC curves against algorithms like EWD, EFD, ID, LD or NDD.

Once you have all your features in the same space, it becomes an usual clustering problem.

Choosing between continuousification or discretization depends on your dataset and what your features look like, so it's a bit hard to say, but I advise you to read the articles I gave you on that topic.

Answer 3

K-means obviously doesn't make any sense, as it computes means (which are nonsensical). Same goes for GMM.

You might want to try distance-based clustering algorithms with appropriate distance functions, for example DBSCAN.

The main challenge is to find a distance function!

While you could put a different distance function into k-means, it will still compute the mean which probably doesn't make much sense (and probably messes with a distance function for discrete values).

Anyway, first focus on define what "similar" is. Then cluster using this definition of similar!

Answer 4

If you are comfortable working with a distance matrix of size num_of_samples x num_of_samples, you could use random forests, as well.

Click here for a reference paper titled Unsupervised learning with random forest predictors.

The idea is creating a synthetic dataset by shuffling values in the original dataset and training a classifier for separating both. During classification you will get an inter-sample distance matrix, on which you could test your favorite clustering algorithm.

Answer 5

Mixed approach to be adopted: 1) Use classification technique (C4.5 decision tree) to classify the data set into 2 classes. 2) Once it is done, leave categorical variables and proceed with continuous variables for clustering.