0
$\begingroup$

I did clustering on a dataset of real-world patients and since the best way to choose the amount of clusters in KMeans clustering is Elbow method and the Silhouette method, I conducted those two and the results are plotted in the graphs below.

The Elbow Method plotted in the graph below enter image description here

The Silhouette Method is plotted in the graph below enter image description here

From multiple blogs, I saw if it is unable to identify the correct amount of clusters correctly using the elbow method, then conduct the Silhouette method. But even by conducting both I find it hard to identify the correct k. A snap of the dataset I have conducted clustering is also added below, enter image description here

Any idea on which is the correct k? Please provide clarification for the answer also since I am afraid of whether I understand them correctly or not. Thanks in advance.

$\endgroup$
  • $\begingroup$ The right k for kmeans depends on your goal. What further analysis do you want to carry out with these clusters? $\endgroup$ – CloseToC Feb 11 at 19:03
  • 1
    $\begingroup$ Before moving on to the question about the number of clusters, did you remember to normalize/standardize the features? $\endgroup$ – Adam Feb 11 at 19:30
0
$\begingroup$

The Elbow Method can sometimes provide a very clear answer, but often it's hard to discriminate the elbow or bend that reveals the best clusters. In the absence of a clear elbow, one can then turn to the Silhouette Method, which involves examining the average Silhouette Coefficient (SC) and finding the number of clusters k that maximize this value (with the understanding that if two values of k are relatively close, there's often a benefit to choosing the simpler model.)

In the case of your diagram for the average SC, it's important to note that the coefficient is only defined for values of 2 and above. Therefore, the highest average SC was present when two clusters were generated, so k = 2 would be your choice using this approach. Having said this, the average SC is still pretty low when k = 2 (just under 0.1), and this led to my question in the comments as to whether you standardized the features before performing K-means clustering.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.