I am working on segmenting my customers with clustering. My dataset size is 7315 rows and 30 features.
So, as a beginner to clustering, I passed all my 29 features (excluding id column) to the cluster.
So, I did the below two approach to find best K
Approach 1 - Ran a for loop and computed the silhouette score to identify optimal K
Approach 2 - Found out distortion score using elbow visualizer
My question is as follows,
a) when elbow visualizer shows that k=6
is the optimal number of clusters, why is my silhouette score less? You can see that silhouette score is high for cluster 2 but for cluster 6, it is low. Is this expected?
I was expecting to see high silhouette score for n_cluster = 6.
b) what is the difference between distortion score and silhouoette score?
You can find details about the code below
Please find below