0
$\begingroup$

I am using:

  • the cluster::pam function in R for clustering
  • distance matrix computed using Gower distance in cluster::daisy in R

The issue I am having is that I run pam 2 times, with a different distance matrix in each run, but I get the same cluster assignment in both runs!!

  • Run 1: Distance matrix based on 13 attributes (11 integers and 2 binary) enter image description here

  • Run 2: Distance matrix based on 13 attributes as above + 1 integer attribute (almost uniform distributed) enter image description here


If I run PAM for each distance matrix I get different average silhouette widths as expected. So far so good. Although it is strange to get such high numbers close to 1!

enter image description here

enter image description here


The problem is the unexpeted results I get when I look at the clustering assignement in each run. Both clustering assignments give the same results! Even though the average widths are different as shown above. Can someone please explain what am I missing. How can they be different? Thanks in advance

enter image description here



EDIT

You can see below also the scatter plot between the 2 distances. Curiously the distance in the 2nd run is one of 3 values (0 or 0.5 or 1). Any ideas why this would happen?

enter image description here

enter image description here

$\endgroup$
6
  • 2
    $\begingroup$ Exactly how different are these distance matrices? You can quickly and easily check by viewing a scatterplot of one against the other. Throwing in one more attribute along with 13 others is unlikely to change things much and perhaps it hasn't made any difference at all. $\endgroup$
    – whuber
    May 28, 2022 at 17:56
  • $\begingroup$ @whuber thnx for the fast reply. So from the descriptive stats on the snapshot above you can see the differences to some degree (median of 0.5 for run 1, compared to a median of 0.3872 for run 2). Also the silhouette widths are different in each run (0.97 for pam on distance matrix of run 1, compared to 0.70 for run 2). So my question is how can the cluster assignments be identical! $\endgroup$ May 28, 2022 at 18:00
  • 2
    $\begingroup$ The descriptive stats are almost worthless. Of course the new distances will differ from the old ones. But if they scarcely change on a relative basis, one would expect the two solutions to be close, if not identical. That's why you need to examine your distance matrices. $\endgroup$
    – whuber
    May 28, 2022 at 18:08
  • $\begingroup$ @whuber I will do the scatter plot and update accordingly. But even if there are minimal differences, how do we explain the big difference in the average silhouette width that still results in the same cluster assignment? $\endgroup$ May 28, 2022 at 18:31
  • 1
    $\begingroup$ Another thing is that the fact that the number of objects in the three clusters is the same doesn't automatically imply that the clusters are the same. It may well be that they are, but to demonstrate this clearly you'd need to table the cluster memberships against each other rather than just giving the numbers of elements. $\endgroup$ May 28, 2022 at 22:24

1 Answer 1

1
$\begingroup$

What I believe is going on here is the following. I assume it's dist1 that can only take three values, not dist2, as opposed to what is shown in the last paragraph (dist1 and dist2 seem to have changed names there). I abbreviate "gower_dist_test" as "dist".

Average Silhouette Width (ASW) of 1 can only be achieved if all observations within a cluster have distance zero between them. This is possible with dist1, as there are many zeroes. Chances are the clusters in the clustering of dist1 are very strong in the sense that they have almost all within-cluster distances zero, and most if not all between-cluster distances as 0.5 or 1. This produces very high ASW values. dist2 looks like dist1 with some "noise" added. Chances are the added variable has relatively little influence on the overall distance other than making the separation between zero, 0.5, 1 distances more "noisy" and less clear. This causes the lower ASW values. Now it is probably something of a coincidence that the resulting clustering is exactly the same, however if the variables defining dist1 define a very strong and clear clustering structure and what is added to arrive at dist2 is just unstructured noise, depending on the "size" of the noise one should expect to see a very similar clustering structure for dist2, and it is not super-unlikely that it's exactly the same.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.