I am trying to clusters points in a high-dimensional space (5000 features for each data point). Each feature can take 0 or 1 value. Also for each point only a small subset of the features will be 1 (sparse data). I don't know ground (truth) labels for these points. I am trying various clustering algorithms (Agglomerative, DBSCAN, kmeans) and distance metrics (jacarrd, cosine, euclidean) from sklearn. One question I face is how to decide the right number of clusters. The approach I am using is to compute silhouette score for different values of num_clusters (or eps in case of DBSCAN) and choose the number of clusters for which silhouette score is maximum.
I see on my dataset that silhouette score is maximum for num_clusters=2. As the num_clusters increase the silhouette score decreases. This holds true for Aggolomerative clustering with jaccard/cosine distance or kmeans as well. My expectation was to have more than two clusters. My questions:
- Is my use of silhouette score correct in determining number of clusters? Are there other scores that might be better suited (which can be applied without knowing ground truth).
- How to determine if I am running into curse of dimensionality? The number of points in my dataset is small less than 1000 (each point has 5000 features). If I compute pairwise distances what should look for to confirm dimensionality problem?
- If it is high dimensionality issue, then what are some options for dimensionality reduction given this is sparse binary data?
- Are there any transformations I should consider applying on this data before doing clustering. Does it make sense to apply tf-idf transformation on this sparse binary data (this is not text document data) and then apply clustering?.
