I am trying to cluster tweets using k means algorithm. In order to find the best number of clusters I run the elbow method and the Silhouette method for 1 to 14 clusters. However, the elbow method gives 8 clusters and the Silhouette method gives 14 clusters which is the max number of clusters (So I suppose that if I check for more clusters the Silhouette score will improve more). The Silhouette output is this:

[(14, 0.01099081385755934), (13, 0.010913600577875339), (15, 0.010735258452729449), (12, 0.009972398884037042), (11, 0.009873906887295263), (10, 0.008924420881453054), (9, 0.00837275406765135), (8, 0.008038285782741293), (7, 0.007561470837121052), (6, 0.006995317289821658), (5, 0.006343859527220237), (4, 0.005040021066862907), (3, 0.004212974332611302), (2, 0.003538406316643856)]

where the first number is the number of clusters and the second is the Silhouette score. I use tfidfVectorizer to tranform the tweets into a tweets x documents matrix. Why is there such a big difference between the 2 methods outputs and which method should I trust?


0.01 is not a good Silhouette.

Essentially the Silhouette says: all of these are bad clusterings.

You need to go back, and improve preprocessing of your data. Maybe also consider other algorithms. Because so far, you have not found reliable clusters. Which is very typical for clustering Tweets: there are a few duplicates, but no clusters that you could find with k-means.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.