# Choosing the best clustering algorithm and evaluating the results

I'm trying to separate my data into clusters using the k-means algorithm and the hierarchical algorithm, choose which algorithm fits my data the best, and evaluate the results. However, all of my results are conflicting.

As a first step, I plotted my data using the k-means method. When I split the data into 6 or more clusters, the clusters overlapped. I decided that 5 clusters is the optimal number in this case, based on the graphs (we want all of the data points within one cluster to have approximately the same distance from the cluster's centre). The Elbow plot showed that the optimal number of clusters is 3, while the Silhouette plot showed that the optimal number of clusters in this case is 5.

Then, I plotted my data using different hierarchical methods (agnes, diana, ward, ward.2 etc). Honestly, I couldn't deduct anything from the respective dendrograms. The Elbow and Silhouette plots show that the optimal number of clusters n this case (FUN = hcut) is 3.

Finally, I used the clValid function for the clustering methods "kmeans", "hierarchical", "agnes", "diana", and for 2 to 7 clusters. These were the optimal score results:

Optimal Scores:

Score  Method       Clusters
Connectivity 9.5071 kmeans       2
Dunn         0.4821 hierarchical 5
Silhouette   0.3050 diana        4


I have to idea how to evaluate my results and choose the most fitting algorithm. Not only are all of the results vastly different, they are not good at all. In all of the cases above, the Silhouette coefficient is about 0.3. The connectivity and Dunn scores are not good either. I know that I shouldn't set my mind on choosing only one algorithm, however these are my assignment's instructions. I also know that hierarchical clustering is not the best for big datasets, however I'm not sure if my dataset qualifies as big (7 variables x 30 rows).

Any help would be appreciated!

Here is my data (it has been scaled):

cities_struct <-
structure(
c(
-0.531504494839432,
0.934498212505445,
-0.482512116444874,-0.601224417939382,
-0.961129966914795,
-0.900831655044569,
-0.968667255898573,
0.849703711437939,
-0.380758715163867,
0.420078239362577,
-0.139565467682961,
1.18511307121607,
-0.139565467682961,
-0.900831655044569,
2.20641572851803,
0.146851513700614,
2.09900936049919,
0.930729568013556,
-1.29465500444698,-0.495702372166485,
0.69330496502454,
-1.11941303557414,
1.73721948927783,-0.900831655044569,
-1.08172659065525,
-0.955477000176961,
-0.424098126820592,
0.744181665665043,
-0.450478638263816,
0.781868110583935,
-0.695243776334721,-0.910447510783003,
-1.41435207817665,
-1.29273817418867,
-1.20760844139709,-1.63907342250225,
-0.246858599892108,
1.11627463698178,
-1.04792409789984,-0.961208096795374,
-0.330402064370802,
0.454800750508065,
0.72658138811597,-0.33304584489228,
0.169272454188477,
1.34363976182886,
1.16280517415978,-0.901458657009979,
-0.211960697008603,
-0.0654952561187398,-0.0395862070082589,
0.983028098699304,
0.173502503022841,
1.11151583204312,-1.41540959038524,
0.784215803484183,
1.22202585784088,
0.702258607318375,
0.716535022134355,
2.04635662443762,
-1.28905720128345,
0.202857461317712,-0.929495034397562,
-1.19245841017978,
-1.57885357459447,
-0.720197653672938,-1.54665397755992,
0.063325874167963,
-0.0225397179241905,
0.202857461317712,-0.0225397179241905,
0.063325874167963,
1.57134033528641,
-1.54665397755992,
1.201044969389,
0.846849402008864,
0.562419628203605,
0.562419628203605,
0.524853431663288,
-0.16207130507394,
0.562419628203605,
-0.489433874925275,
0.562419628203605,
-0.908028636374524,
-0.628965462075025,
-0.730930852684457,
0.23505705835227,
0.562419628203605,
1.56060713627489,
2.48366225126554,
1.41192357030813,
1.2940176354181,
-0.725121499573696,
-0.747228862365577,-0.563000839099902,
-0.865134797255608,
0.173911253962797,
2.61309028200033,-0.0913770995397747,
0.0928509237259,
-0.415618420487362,
-0.106115341401029,-0.636692048406172,
-1.52835568101204,
-0.673537653059307,
1.38981620751625,-0.0545314948866398,
-1.04936282052128,
-0.526155234446767,
1.15400433773619,-0.437725783279243,
-1.29254381123197,
0.085481802795273,
0.741333565621075,-1.1893761182032,
1.49298390054503,
-0.887242160047489,
0.947668951678631,
0.675011477245432,
-0.282974243736076,
-0.396322302114211,
-0.360792185380256,-0.422162387011633,
-0.68702325721021,
-0.735473416392876,
-0.873717870594084,
0.0436051432643998,
0.440896448562264,
-0.564282853947455,
-0.548132800886566,-0.00355301167339554,
0.53585876056029,
-0.82203770079924,
-0.71738535696468,-0.802011635003738,
-0.469966544071864,
-0.14373547224191,
-0.693483278434565,-0.647617127741641,
2.6747717879444,
-0.220609724811741,
-0.435082429460344,-0.407304338195616,
1.74517473375964,
-0.725783384556343,
3.07206309324227,
0.474488558928913,
0.917000012797266,
0.99775027810171,
-0.22513173966879,
0.886213890432966,
0.204510897792223,
0.917200390098455,
0.576348893778083,
1.30453163591706,
0.390429895785153,
0.312963646621432,
-0.244793347357358,-0.415219095517544,
0.343950146286921,
-0.3997258456848,
-0.0743675991971721,-1.17438833732201,
0.932693639931199,
0.436909645283385,
-1.85609132996275,
0.917200390098455,
1.30453163591706,
0.390429895785153,
-0.880016590499869,
0.312963646621432,
0.343950146286921,
-1.68566558180257,
-1.74763858113354,-0.3997258456848,
-0.0743675991971721,
-1.17438833732201,
1.97074137872506,
0.436909645283385,
-1.85609132996275,
-0.206505199008913,
0.365459487675157,-0.981534271497978,
-0.00259471159639672,
-0.407031279991346,-0.178583845960726,
-0.950228512019708,
0.0388642671721227,
-0.533100419511947,-0.038130979112271,
-0.397724162308617,
-0.236964856879662,
0.481375407905508,
0.141242561682141,
1.77421866419731,
-1.72356538129373,
-0.981534271497978,-0.407031279991346,
-0.178583845960726,
1.56776987196223,
-0.950228512019708,-0.038130979112271,
0.108244598988829,
1.07364653317008,
-0.397724162308617,-0.236964856879662,
0.481375407905508,
2.76331144338914,
1.77421866419731,-1.72356538129373
),
dim = c(30L, 7L),
dimnames = list(
c(
"1",
"2",
"3",
"4",
"5",
"6",
"8",
"9",
"10",
"11",
"15",
"16",
"18",
"19",
"21",
"22",
"25",
"27",
"28",
"29",
"30",
"33",
"34",
"35",
"37",
"38",
"40",
"43",
"44",
"45"
),
c(
"sunshine",
"pollution",
"work",
"activities",
"places",
"bottle",
"membership"
)
),
"scaled:center" = c(
sunshine = 2140.06666666667,
pollution = 44.0786666666667,
work = 1674.2,
activities = 230.4,
places = 1661.5,
bottle = 1.348,
membership = 37.3406666666667
),
"scaled:scale" = c(
sunshine = 530.694790740912,
pollution = 18.9123112126017,
work = 186.337735641866,
activities = 135.701396328546,
places = 1547.98253019635,
bottle = 0.645442377032191,
membership = 11.818911477194
)
)

• 7x30 is a tiny dataset.
– Christian Sloper
Commented Jan 8, 2023 at 15:58
• Usually you'd want about 20-30 examples of each cluster to perform a decent clustering analysis. I don't think it makes any sense to evaluate whether a dataset of 30 individuals total can be split into 10 clusters. The variability in the result will be likely be too high to be useful, unless the clusters are clearly very different from one another. There's a good chance you'd get a totally different result running the exact same algorithm on a different sampling of the same population. Commented Sep 5 at 14:15