I have used the R package dtwclust to generate clusters for more than a thousand time-series objects.Since I did not have any prior information on the number or validity of clusters, I used a suite of internal CVIs (Cluster Validity Indices) implemented in the same package to compare across 4 different clustering methods, as well as to find the optimum value of k (number of clusters). I tested 6 CVIs:
- "Sil" (!): Silhouette index (Rousseeuw (1987); to be maximized)
- "D" (!): Dunn index (Arbelaitz et al. (2013); to be maximized).
- "COP" (!): COP index (Arbelaitz et al. (2013); to be minimized).
- "DB" (?): Davies-Bouldin index (Arbelaitz et al. (2013); to be minimized).
- "DBstar" (?): Modified Davies-Bouldin index (DB*) (Kim and Ramakrishna (2005); to be minimized).
- "CH" (~): Calinski-Harabasz index (Arbelaitz et al. (2013); to be maximized).
- "SF" (~): Score Function (Saitta et al. (2007); to be maximized)
across k = 2 to 30 clusters, for 4 crips-clustering algorithms:
- partitional k-means with Euclidean distance,
- partitional k-means with Dynamic Time Warping distance,
- Heirarchical agglomerative with Euclidean distance, and
- Heirarchical agglomerative with Dynamic Time Warping distance.
I obtained the following plots for CVI-metric (y-axis) versus number of clusters (x-axis) for the different CVIs and for different clustering methods.
I am trying to decide on the optimum k + overall best clustering method based on this result. Can someone help me interpret this plot?
I read the resources here: How to select a clustering method? How to validate a cluster solution (to warrant the method choice)? and useful as they are, I do not see a consistent "sharp bend" or "extremum" across methods and indices. For instance, based on SIL and CH: I would say p-DTW and h-DTW outperform other methods. But based on DB, DBstart and DUNN - I would select h-euclidean over every other method.
Similarly for optimum k: I can see sharp bends at k = 12 for both DB and DBstar, but the sharp extremum for DUNN is at k = 14.
I need help to interpret this plot. I do not think the raw data or the codes are needed as part of this query, but if the moderators feel like I should add that, please let me know. Thanks community.