1
$\begingroup$

I'm learning about unsupervised learning and I tried to use KMeans, AgglomerativeClustering and DBSCAN on the same datase. The result was ok, they seems to work fine according silhouette_score() function in which the best score was 0.1935 from AgglomerativeClustering. It found 2 clusters by the way. The problem is I was unable to find the differences between these two clusters. I tried several plots pairing clusters 0 and 1, but the same patterns I find in one cluster I find in the other. So, my question is:

What techniques do you use to identify differences between clusters?

Since I'm learning it seems to me clustering is just part of the problem. You have to understand the differences between the clusters to recommend different products to them for example. So labelling is not enought.

$\endgroup$

1 Answer 1

1
$\begingroup$

The OP is absolutely correct that assigning cluster labels without a way to interpret their real-world relevance is often a pointless exercise.

The simplest technique to identify those differences (aside from doing EDA) is to run a classifier where cluster labels are treated as class membership. Starting off, I would recommend a relatively "straight-forward" algorithm like logistic regression or a moderately-sized decision tree. In that way, the feature importance and interpretation are reasonably straightforward to estimate.

Notably, if the result of this "classification" task is bad, then we have a good idea that the cluster assignment at hand most likely is not very interpretable or well-defined (cause if it was, the learning algorithms would have picked). There are some other metrics like cluster cardinality and cluster magnitude to assess the "quality of clustering" but I cannot recommend them, I have found them often useless and totally arbitrary.

While not commented on in the answer, if the dimensionality of the points we are trying to cluster is "large" (e.g. 10+) the concept of a neighbourhood becomes somewhat ill-defined in which case we should re-work our problem formulation as well as the feature space we are working with. (See issues with curse of dimensionality; CV.Se has some excellent threads on this too: eg. "Why is Euclidean distance not a good metric in high dimensions?", "How do I know my k-means clustering algorithm is suffering from the curse of dimensionality?")

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.