High dimensional clustering of percentage data using cosine similarity

Question

I'm building a clustering algorithm and was trying to determine the best way to get separate and accurate clusters.

I have 300+ features to cluster on, and they are all percentages between 0 and 1. The columns are products and the rows are customers. The matrix values represent the percentage of a given customers "wallet" that is dedicated to each product. So if one were to add up all the values in any given row, the result would be 1.

Because of the high number of features, I chose to opt for Spherical K-Means instead of regular K-Means, as it seems to be more robust to a high number of features. However, I'm now not entirely sure if using cosine similarity as a distance metric for clustering based on percentage data was valid? It's my understanding that cosine similarity is often used with text data, and helps account for varying document lengths through measuring angle similarity rather than raw distance. However, it seems to me that this logic only makes sense if the data is raw counts.

I also have the raw count data at my disposal if needed. There is some separation between the clusters with the percentage data, but I'm curious if the raw count data would likely perform better? I haven't been able to find a definitive answer about clustering on cosine similarity with percentage data and I'm not super knowledgeable about all of this so I wanted to confirm if my concerns were valid.

I've also heard that K-Means isn't always the best algorithm for clustering, and I am aware there are many other out there, such as some soft-clustering algorithms. Is there any models that might come to mind as being more suited for this type of problem?

I'm coding this in python, so it would be preferable if the algorithm was easily implemented in python

Answer 1

There is no reason why cosine would be any better for high-dimensional data than Euclidean.

It's east to prove that cosine is essentially Euclidean after normalization, so the 'difficulty' maybe drops to 299 instead of 300 dimensions...

There are more appropriate distance functions for histograms such as Kullback Leibler Divergence. They cannot be used with k-means.

So I suggest you first identify a good similarity function, then decide which clustering to use with that. There are plenty of choices, so I'm not going to recommend any because I don't have your data.