I have carried out a clustering of coordinate points (longitude, latitude) and found surprising, adverse results from clustering criteria for the optimal number of clusters. The criteria are taken from the
clusterCrit() package. The points which I am trying to cluster on a plot (the geographic characteristics of the data set is clearly visible) :
The full procedure was the following :
- Carried out hierarchical clustering on 10k points and saved medoids for 2 : 150 clusters.
- Took the medoids from (1) as seeds for kmeans clustering of 163k observations.
- Checked 6 different clustering criteria for the optimal number of clusters.
Only 2 clustering criteria gave results that make sense for me – the Silhouette and Davies-Bouldin criteria. For both of them one should look for the maximum on the plot. It seems both give the answer “22 Clusters is a good number”. For the graphs below: on the x axis is the number of clusters and on the y axis the value of the criterion, sorry for the wrong descriptions on the image. Silhouette and Davies-Bouldin respectively :
Now let’s look at Calinski-Harabasz and Log_SS values. The maximum is to be found on the plot. The graph indicates that the higher the value the better the clustering. Such a steady growth is quite surprising, I think 150 clusters is already a quite high number. Below the plots for Calinski-Harabasz and Log_SS values respectively.
Now for the most surprising part the last two criteria. For the Ball-Hall the biggest difference between two clusterings is desired and for Ratkowsky-Lance the maximum. Ball-Hall and Ratkowsky-Lance plots respectively :
The last two criteria give completely adverse answers (the smaller the number of clusters the better) than the 3rd and 4th criteria. How is that possible? For me it seems like only the first two criteria were able to make any sense of the clustering. A Silhouette width of around 0.6 is not that bad. Should I just skip the indicators that give strange answers and believe in those that give reasonable answers?
Edit: Plot for 22 clusters
You can see that the data is quite nicely clustered in 22 groups so criteria indicating that you should choose 2 clusters seem to have weaknesses, the heuristic isn't working properly. It is ok when I can plot the data or when the data can be packed in less than 4 principal components and then plotted. But if not? How should I choose the number of clusters other than by using a criterion? I have seen tests which indicated Calinski and Ratkowsky as very good criteria and still they give adverse results for an seemingly easy data set. So maybe the question shouldn't be "why are the results differing" but "how much can we trust those criteria?".
Why is an euclidian metric not good? I am not really interested in the actual, exact distance between them. I understand the true distance is spheric but for all points A,B,C,D if Spheric(A,B) > Spheric(C,D) than also Euclidian(A,B) > Euclidian(C,D) which should be sufficient for for a clustering metric.
Why I want to cluster those points? I want to build a predictive model and there is a lot of information contained in the location of each observation. For each observation I also have cities and regions. But there are too many different cities and I don't want to make for example 5000 factor variables; therefore I thought about clustering them by coordinates. It worked pretty well as the densities in different regions are different and the algorithm found it, 22 factor variables would be all right. I could also judge the goodness of the clustering by the results of the predictive model but I am not sure if this would be wise computationally. Thanks for the new algorithms, I will definitely try them if they work fast on huge data sets.