I have a set of points that I want to cluster into groups according to a number of features computed.
I have distance matrix containing the distances between all different pairs of points. I have tried K-Means, and DBSCAN first but since I have no clue about that number of clusters underlying in the data, required for K-Means or the optimal Epsilon and MinPnts required for DBSCAN. I decided to turn to Hierarchical Agglomeration Clustering since it doesn't require setting any parameters for clustering.
I used R to perform HAC. Now to turn the resultant dendrogram into a number of groups of points (flat clusters), I want to choose which level to cut the tree at (the same as choosing the number of clusters).
I managed to do this for different values for the number of clusters and evaluate the Silhouette value for each clustering results and therefore I can choose the number of clusters that gives me the maximum silhouette value as the optimal number of clusters.
What happens is that I find that the Maximum silhouette value (0.8) obtained is for a number of clusters = 5 but the cluster sizes are not very good (one cluster is > 900 points, second one is 5 points, and the other three are single point for each). I was using group average for the HAC cluster-cluster distance..
I tried also Ward's method (which is supposed to give clusters of equal sizes) but although this one give me somehow better distribution for the number of points in each cluster... The silhouette value for the clusters produced is really low (0.1).
Do you suggest any good methods for doing clustering without requiring parameters? or to obtain the optimal values for clustering parameters (dendrogram cutting level for HAC, (epsilon, MinPnts) for DBSCAN)?