0
$\begingroup$

I am doing a cluster analysis with agglomerative hierarchical clustering on my asymmetrical binary data. For finding the number of clusters, I tried all three of the most mentioned methods (Elbow, Silhouette, and Gap stat.); however the results are not overlapping and in the case of Elbow, the visual does not look like there is a clear point (See the photo). Any suggestion on what could be an optimal k?

enter image description here

enter image description here

Improve this question
$\endgroup$
10
  • $\begingroup$ Since your data are binary or categorical, it is unwise to use clustering methods (what linkage did you use?) and clustering validity criteria that are based on centroids and deviations from them. So, you better dismiss Gap and Elbow. As your Silhouette suggest, as it is low, you probably don't have any clusters at all. $\endgroup$
    – ttnphns
    Commented Nov 5, 2021 at 8:07
  • $\begingroup$ A pair of links for you to read. stats.stackexchange.com/q/195456/3277, stats.stackexchange.com/q/21807/3277 $\endgroup$
    – ttnphns
    Commented Nov 5, 2021 at 8:13
  • $\begingroup$ Thanks for your comment @ttnphns! I used Ward.D for the dissimilarity matrix calculated with Jaccard dist. measure (my data is asymmetrical binary (0-absence, 1-presence) where I encoded my categorical data). I was also referring to different sources as well as literature which supported the choice. $\endgroup$
    – user339807
    Commented Nov 5, 2021 at 8:37
  • $\begingroup$ No, you are not warranted to use Ward with proximities other than euclidean distance. stats.stackexchange.com/q/195446/3277 $\endgroup$
    – ttnphns
    Commented Nov 5, 2021 at 8:40
  • $\begingroup$ @ttnphns, any suggestion (reference) for the linkage method for asymmetrical binary data? $\endgroup$
    – user339807
    Commented Nov 5, 2021 at 8:45

1 Answer 1

Reset to default
0
$\begingroup$
  • All those methods use different criteria, hence it's nothing strange that they can diverge. The methods already provide you some choices, its up to you to make the decision.
  • Elbow does not give any clear answer and it's interpretation is subjective.
  • In most cases, you shouldn't make the decision based only on the metrics. Compare the solutions with two or three clusters, which one makes more sense for the data? Which one is easier to interpret?
  • If you want a less subjective criteria, you can use model-based clustering, where likelihood-ratio tests or criteria such as AIC can be used to distinguish between the solutions.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.