9
$\begingroup$

I frequently come across data sets that have both categorical and numeric data. I think this is just a fact of life where the data is not all in one category. I'm basically trying to find some practical approaches or strategies on how to go about solving this problem.

In many books and initial searches on google I tend to get some sort of Kmeans clustering and a lot of phd-looking papers. I think this is a common problem so I'm wondering if anyone can guide me to some practical suggestions or methods that I can further investigate and implement in R or Python?

I'm basically lost and just looking for some direction!

$\endgroup$
  • $\begingroup$ Easy Way: Any categorical data can be handle as numeric using one hot encoding. This is implemented in sklearn $\endgroup$ – Romain Dec 20 '15 at 5:25
  • 1
    $\begingroup$ K-means should not be used in the presence of categorical data. What is "mean" of such data, after all? Search this site for clustering categorical data, mixed-type data, binary data. $\endgroup$ – ttnphns Dec 20 '15 at 8:48
  • $\begingroup$ @Romain it can be handled this way, but the results will be meaningless. $\endgroup$ – Anony-Mousse Dec 20 '15 at 11:20
  • $\begingroup$ Note that the "duplicate" question in the text asks about "mixed" type data, too. For Gower, see also: stats.stackexchange.com/q/15287/7828 and @ttnphns answer there $\endgroup$ – Anony-Mousse Dec 20 '15 at 11:27
  • $\begingroup$ k-prototypes might be an appropriate technique. More details you can see here github.com/nicodv/kmodes $\endgroup$ – prashanth Oct 6 '17 at 9:48
8
$\begingroup$

Distance-based clustering algorithms can handle categorical data

You only have to choose an appropriate distance function such as Gower's distance that combines the attributes as desired into a single distance. Then you can run Hierarchical Clustering, DBSCAN, OPTICS, and many more.

Sounds good, but it is only part of the story - your choice of distance function has a massive impact on your results.

Results will probably never be "sound" with categorical data

Nevertheless, clustering may end up never to work well on such data. Consider the description from Wikipedia:

Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters).

So for clustering, you need a qualitative similarity, so the algorithm knows when objects are "more similar" than others.

That is why many algorithms use some form of distance: closer = more similar. It is a very intuitive way of qualifying similarity.

With continuous variables, it is challenging enough to properly normalize the data. Most people either ignore data normalization, normalize to $[0;1]$ or standardize to $\mu=0$, $\sigma=1$. With high-dimensional data, people sometimes also do PCA (but more often than not use it in an absurd way, witjout considering the effect this has on their data). The good thing with continuous variables is that they can be quite "forgiving". If your scaling/weighting is a little off, the outcomes may still be good. Similarly, if there is a small error in your data, it only has a small effect on your distance.

Unfortunately, this does not carry over to discrete, likert, or categorical variables. There are plenty of approaches used, such as one-hot encoding (every category becomes its own attribute), binary encodings (first category is 0,0; second is 0,1, third is 1,0, fourth is 1,1) that effectively map your data in a $\mathbb{R}^{d}$ space, where you could use k-means and all that. But these approaches are highly fragile. They tend to work if you have only binary categories, unless they vary too much in frequency. But the problem is that you have low discriminiability. You may have 0 objects at distance 0 (these would be duplicates), then nothing for a while, and then hundreds of objects at distance 2. But nothing inbetween. So whichever algorithm you use, it will have to merge all these objects at once, because they have the exact same similarity. In the worst case, your data might go from duplicates-only to everything-is-one-cluster because of this. Now if you would put different weight on every attribute this will be slightly better (you will still habe lots of object pairs that differ only in this one attribute, and thus have the same distance) but how do you choose the weights of attributes? There does not appear a statistically sound unsupervised way.

So in conclusion, I believe that categorical data does not cluster in the way clustering is commonly defined because the discrete nature yields too little discrimination/ranking of similarities. It may have frequent patterns as detected e.g. by Apriori, but that is a very different definition. And how to combine these two is not obvious. So for categorical data, I recommend frequent patterns. These make much more sense than "clusters".

$\endgroup$
1
$\begingroup$

Check out the R package ClusterOfVar. It handles mixed data.

Edit: figured I should mention that k-means isn't actually the best clustering algorithm. It prefers even density, globular clusters, and each cluster has roughly the same size. If those are violated then K-means probably won't perform well. It is used often because you can use a different objective function to apply to many different situations, such as using cos() for high dimensional data.

I don't have any experience with Python for clustering, but I've heard the R package I mentioned above is pretty good and incorporates good algorithms.

$\endgroup$
  • $\begingroup$ Awesome I really appreciate this. Any other advice on best clustering methods? I often hear about 'clustering' but the reading material or implementation of it always seems to be much more 'dense' in comparison to how/why you would use GLMs, Randoms Forests, SVM, etc. Or at least that is my impression. $\endgroup$ – user3788557 Dec 20 '15 at 7:49
  • $\begingroup$ As I understand you are talking about hclustvar() from package ClusterOfVar. Again as I understand, for categorical variables it clusters variables, not categories of variables. $\endgroup$ – vasili111 Oct 19 at 2:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.