1
$\begingroup$

I asked a number of 71 'experts' to sort 92 different psychological constructs based on their similarity. Based on their answers, I constructed a dissimilarity matrix.

Initially, I wanted to analyse the data using hierarchical cluster analysis (HCA). Eventually, however, I decided to go for partitioning around medoids (PAM). The main reason has been that PAM allows me to identify a representative example for each cluster determined. It basically allows me to label those clusters by the means of these examples. In my next phase of my research, I want to question these experts on these representative examples.

I run PAM in R using the cluster package. As for the number of clusters, I created a loop testing the solution for each number of clusters from $k=2...91$. Using the average silhouette width as the benchmark, I determined the optimal number of clusters:

enter image description here

As one may be able to see in the graph, the optimal cluster number is $2$ (I also confirmed this using pamk from fpc package. I am somewhat "disappointed" about the optimal number of 2 clusters. Since I plan to interview experts about each of the clusters (what is it? what are the effects etc.? I hoped to get a larger number of clusters.

How common and "justifiable" is it to deviate from optimal cluster number that has been statistically derived for PAM applications? For example, around 43 there is another "elbow". If I decided to go for this, to what extent would this be unfounded?

$\endgroup$
1
$\begingroup$

Intrinsic measures such as Silhouette are only heuristcs.

You don't look for "elbows" in Silhouette... You look for the maximum.

All your Silhouette Scores are very low. All failed, probably your dissimilarities were already too bad in quality.

But feel free to use a larger k if that is more interesting to you.

Don't assume the clusters to be correct either. There is no "correct". They can be interesting to a user, or not.

$\endgroup$
0
$\begingroup$

You can try other similar clustering methods that associate every element in your dataset to an exemplar: I would advise AP clustering (there is an R package for that called "apcluster").

I work on this exact problem, and was wondering if you could share your dataset? It would help me a lot.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.