2
$\begingroup$

I am trying to clusters points in a high-dimensional space (5000 features for each data point). Each feature can take 0 or 1 value. Also for each point only a small subset of the features will be 1 (sparse data). I don't know ground (truth) labels for these points. I am trying various clustering algorithms (Agglomerative, DBSCAN, kmeans) and distance metrics (jacarrd, cosine, euclidean) from sklearn. One question I face is how to decide the right number of clusters. The approach I am using is to compute silhouette score for different values of num_clusters (or eps in case of DBSCAN) and choose the number of clusters for which silhouette score is maximum.

I see on my dataset that silhouette score is maximum for num_clusters=2. As the num_clusters increase the silhouette score decreases. This holds true for Aggolomerative clustering with jaccard/cosine distance or kmeans as well. My expectation was to have more than two clusters. My questions:

  • Is my use of silhouette score correct in determining number of clusters? Are there other scores that might be better suited (which can be applied without knowing ground truth).
  • How to determine if I am running into curse of dimensionality? The number of points in my dataset is small less than 1000 (each point has 5000 features). If I compute pairwise distances what should look for to confirm dimensionality problem?
  • If it is high dimensionality issue, then what are some options for dimensionality reduction given this is sparse binary data?
  • Are there any transformations I should consider applying on this data before doing clustering. Does it make sense to apply tf-idf transformation on this sparse binary data (this is not text document data) and then apply clustering?.
$\endgroup$
1
$\begingroup$

With the high dimensionality, silhouette may become hard to use. Don't just look for the maximum - if the maximum is small, you may still have a pretty bad result.

If you could add a dendrogram to your question, it could be easier to talk about the number of clusters.

Such data is usually quite noisy, and you need an algorithm that is robust to outliers. So k-means will probably not work well.

Usually, on high-dimensional binary data, one would use frequent itemsets and biclustering.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.