What is the best k in kmeans clustering [duplicate]

Question

I did clustering on a dataset of real-world patients and since the best way to choose the amount of clusters in KMeans clustering is Elbow method and the Silhouette method, I conducted those two and the results are plotted in the graphs below.

The Elbow Method plotted in the graph below

The Silhouette Method is plotted in the graph below

From multiple blogs, I saw if it is unable to identify the correct amount of clusters correctly using the elbow method, then conduct the Silhouette method. But even by conducting both I find it hard to identify the correct k. A snap of the dataset I have conducted clustering is also added below,

Any idea on which is the correct k? Please provide clarification for the answer also since I am afraid of whether I understand them correctly or not. Thanks in advance.

Answer 1

The Elbow Method can sometimes provide a very clear answer, but often it's hard to discriminate the elbow or bend that reveals the best clusters. In the absence of a clear elbow, one can then turn to the Silhouette Method, which involves examining the average Silhouette Coefficient (SC) and finding the number of clusters k that maximize this value (with the understanding that if two values of k are relatively close, there's often a benefit to choosing the simpler model.)

In the case of your diagram for the average SC, it's important to note that the coefficient is only defined for values of 2 and above. Therefore, the highest average SC was present when two clusters were generated, so k = 2 would be your choice using this approach. Having said this, the average SC is still pretty low when k = 2 (just under 0.1), and this led to my question in the comments as to whether you standardized the features before performing K-means clustering.