Clustering with categorical and numeric data [duplicate]

Question

I frequently come across data sets that have both categorical and numeric data. I think this is just a fact of life where the data is not all in one category. I'm basically trying to find some practical approaches or strategies on how to go about solving this problem.

In many books and initial searches on google I tend to get some sort of Kmeans clustering and a lot of phd-looking papers. I think this is a common problem so I'm wondering if anyone can guide me to some practical suggestions or methods that I can further investigate and implement in R or Python?

I'm basically lost and just looking for some direction!

Answer 1

Distance-based clustering algorithms can handle categorical data

You only have to choose an appropriate distance function such as Gower's distance that combines the attributes as desired into a single distance. Then you can run Hierarchical Clustering, DBSCAN, OPTICS, and many more.

Sounds good, but it is only part of the story - your choice of distance function has a massive impact on your results.

Results will probably never be "sound" with categorical data

Nevertheless, clustering may end up never working well on such data. Consider the description from Wikipedia:

Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters).

So for clustering, you need a qualitative similarity, so the algorithm knows when objects are "more similar" than others.

That is why many algorithms use some form of distance: closer = more similar. It is a very intuitive way of qualifying similarity.

With continuous variables, it is challenging enough to properly normalize the data. Most people either ignore data normalization, normalize to $[0;1]$ or standardize to $\mu=0$, $\sigma=1$. With high-dimensional data, people sometimes also do PCA (but more often than not use it in an absurd way, without considering the effect this has on their data). The good thing with continuous variables is that they can be quite "forgiving". If your scaling/weighting is a little off, the outcomes may still be good. Similarly, if there is a small error in your data, it only has a small effect on your distance.

Unfortunately, this does not carry over to discrete, likert, or categorical variables. There are plenty of approaches used, such as one-hot encoding (every category becomes its own attribute), binary encodings (first category is 0,0; second is 0,1, third is 1,0, fourth is 1,1) that effectively map your data in a $\mathbb{R}^{d}$ space, where you could use k-means and all that. But these approaches are highly fragile. They tend to work if you have only binary categories unless they vary too much in frequency. But the problem is that you have low discriminability. You may have 0 objects at distance 0 (these would be duplicates), then nothing for a while, and then hundreds of objects at distance 2. But nothing in between. So whichever algorithm you use, it will have to merge all these objects at once, because they have the exact same similarity. In the worst case, your data might go from duplicates-only to everything-is-one-cluster because of this. Now if you would put different weights on every attribute this will be slightly better (you will still have lots of object pairs that differ only in this one attribute, and thus have the same distance) but how do you choose the weights of attributes? There does not appear a statistically sound unsupervised way.

So in conclusion, I believe that categorical data does not cluster in the way clustering is commonly defined because the discrete nature yields too little discrimination/ranking of similarities. It may have frequent patterns as detected e.g. by Apriori, but that is a very different definition. And how to combine these two is not obvious. So for categorical data, I recommend frequent patterns. These make much more sense than "clusters".

Answer 2

Check out the R package ClusterOfVar. It handles mixed data.

Edit: figured I should mention that k-means isn't actually the best clustering algorithm. It prefers even density, globular clusters, and each cluster has roughly the same size. If those are violated then K-means probably won't perform well. It is used often because you can use a different objective function to apply to many different situations, such as using cos() for high dimensional data.

I don't have any experience with Python for clustering, but I've heard the R package I mentioned above is pretty good and incorporates good algorithms.