Choosing the best clustering algorithm and evaluating the results

Question

I'm trying to separate my data into clusters using the k-means algorithm and the hierarchical algorithm, choose which algorithm fits my data the best, and evaluate the results. However, all of my results are conflicting.

As a first step, I plotted my data using the k-means method. When I split the data into 6 or more clusters, the clusters overlapped. I decided that 5 clusters is the optimal number in this case, based on the graphs (we want all of the data points within one cluster to have approximately the same distance from the cluster's centre). The Elbow plot showed that the optimal number of clusters is 3, while the Silhouette plot showed that the optimal number of clusters in this case is 5.

Then, I plotted my data using different hierarchical methods (agnes, diana, ward, ward.2 etc). Honestly, I couldn't deduct anything from the respective dendrograms. The Elbow and Silhouette plots show that the optimal number of clusters n this case (FUN = hcut) is 3.

Finally, I used the clValid function for the clustering methods "kmeans", "hierarchical", "agnes", "diana", and for 2 to 7 clusters. These were the optimal score results:

Optimal Scores:

             Score  Method       Clusters
Connectivity 9.5071 kmeans       2       
Dunn         0.4821 hierarchical 5       
Silhouette   0.3050 diana        4 

I have to idea how to evaluate my results and choose the most fitting algorithm. Not only are all of the results vastly different, they are not good at all. In all of the cases above, the Silhouette coefficient is about 0.3. The connectivity and Dunn scores are not good either. I know that I shouldn't set my mind on choosing only one algorithm, however these are my assignment's instructions. I also know that hierarchical clustering is not the best for big datasets, however I'm not sure if my dataset qualifies as big (7 variables x 30 rows).

Any help would be appreciated!

Here is my data (it has been scaled):

cities_struct <-
  structure(
    c(
      -0.531504494839432,
      0.934498212505445,
      -0.482512116444874,-0.601224417939382,
      -0.961129966914795,
      -0.900831655044569,
      -0.968667255898573,
      0.849703711437939,
      -0.380758715163867,
      0.420078239362577,
      -0.139565467682961,
      1.18511307121607,
      -0.139565467682961,
      -0.900831655044569,
      2.20641572851803,
      0.146851513700614,
      2.09900936049919,
      0.930729568013556,
      -1.29465500444698,-0.495702372166485,
      0.69330496502454,
      -1.11941303557414,
      1.73721948927783,-0.900831655044569,
      -1.08172659065525,
      -0.955477000176961,
      -0.424098126820592,
      0.744181665665043,
      -0.450478638263816,
      0.781868110583935,
      -0.695243776334721,-0.910447510783003,
      -1.41435207817665,
      -1.29273817418867,
      -1.20760844139709,-1.63907342250225,
      -0.246858599892108,
      1.11627463698178,
      -1.04792409789984,-0.961208096795374,
      -0.330402064370802,
      0.454800750508065,
      0.72658138811597,-0.33304584489228,
      0.169272454188477,
      1.34363976182886,
      1.16280517415978,-0.901458657009979,
      -0.211960697008603,
      -0.0654952561187398,-0.0395862070082589,
      0.983028098699304,
      0.173502503022841,
      1.11151583204312,-1.41540959038524,
      0.784215803484183,
      1.22202585784088,
      0.702258607318375,
      0.716535022134355,
      2.04635662443762,
      -1.28905720128345,
      0.202857461317712,-0.929495034397562,
      -1.19245841017978,
      -1.57885357459447,
      -0.720197653672938,-1.54665397755992,
      0.063325874167963,
      -0.0225397179241905,
      0.202857461317712,-0.0225397179241905,
      0.063325874167963,
      1.57134033528641,
      -1.54665397755992,
      1.201044969389,
      0.846849402008864,
      0.562419628203605,
      0.562419628203605,
      0.524853431663288,
      -0.16207130507394,
      0.562419628203605,
      -0.489433874925275,
      0.562419628203605,
      -0.908028636374524,
      -0.628965462075025,
      -0.730930852684457,
      0.23505705835227,
      0.562419628203605,
      1.56060713627489,
      2.48366225126554,
      1.41192357030813,
      1.2940176354181,
      -0.725121499573696,
      -0.747228862365577,-0.563000839099902,
      -0.865134797255608,
      0.173911253962797,
      2.61309028200033,-0.0913770995397747,
      0.0928509237259,
      -0.415618420487362,
      -0.106115341401029,-0.636692048406172,
      -1.52835568101204,
      -0.673537653059307,
      1.38981620751625,-0.0545314948866398,
      -1.04936282052128,
      -0.526155234446767,
      1.15400433773619,-0.437725783279243,
      -1.29254381123197,
      0.085481802795273,
      0.741333565621075,-1.1893761182032,
      1.49298390054503,
      -0.887242160047489,
      0.947668951678631,
      0.675011477245432,
      -0.282974243736076,
      -0.396322302114211,
      -0.360792185380256,-0.422162387011633,
      -0.68702325721021,
      -0.735473416392876,
      -0.873717870594084,
      0.0436051432643998,
      0.440896448562264,
      -0.564282853947455,
      -0.548132800886566,-0.00355301167339554,
      0.53585876056029,
      -0.82203770079924,
      -0.71738535696468,-0.802011635003738,
      -0.469966544071864,
      -0.14373547224191,
      -0.693483278434565,-0.647617127741641,
      2.6747717879444,
      -0.220609724811741,
      -0.435082429460344,-0.407304338195616,
      1.74517473375964,
      -0.725783384556343,
      3.07206309324227,
      0.474488558928913,
      0.917000012797266,
      0.99775027810171,
      -0.22513173966879,
      0.886213890432966,
      0.204510897792223,
      0.917200390098455,
      0.576348893778083,
      1.30453163591706,
      0.390429895785153,
      0.312963646621432,
      -0.244793347357358,-0.415219095517544,
      0.343950146286921,
      -0.3997258456848,
      -0.0743675991971721,-1.17438833732201,
      0.932693639931199,
      0.436909645283385,
      -1.85609132996275,
      0.917200390098455,
      1.30453163591706,
      0.390429895785153,
      -0.880016590499869,
      0.312963646621432,
      0.343950146286921,
      -1.68566558180257,
      -1.74763858113354,-0.3997258456848,
      -0.0743675991971721,
      -1.17438833732201,
      1.97074137872506,
      0.436909645283385,
      -1.85609132996275,
      -0.206505199008913,
      0.365459487675157,-0.981534271497978,
      -0.00259471159639672,
      -0.407031279991346,-0.178583845960726,
      -0.950228512019708,
      0.0388642671721227,
      -0.533100419511947,-0.038130979112271,
      -0.397724162308617,
      -0.236964856879662,
      0.481375407905508,
      0.141242561682141,
      1.77421866419731,
      -1.72356538129373,
      -0.981534271497978,-0.407031279991346,
      -0.178583845960726,
      1.56776987196223,
      -0.950228512019708,-0.038130979112271,
      0.108244598988829,
      1.07364653317008,
      -0.397724162308617,-0.236964856879662,
      0.481375407905508,
      2.76331144338914,
      1.77421866419731,-1.72356538129373
    ),
    dim = c(30L, 7L),
    dimnames = list(
      c(
        "1",
        "2",
        "3",
        "4",
        "5",
        "6",
        "8",
        "9",
        "10",
        "11",
        "15",
        "16",
        "18",
        "19",
        "21",
        "22",
        "25",
        "27",
        "28",
        "29",
        "30",
        "33",
        "34",
        "35",
        "37",
        "38",
        "40",
        "43",
        "44",
        "45"
      ),
      c(
        "sunshine",
        "pollution",
        "work",
        "activities",
        "places",
        "bottle",
        "membership"
      )
    ),
    "`scaled:center`" = c(
      sunshine = 2140.06666666667,
      pollution = 44.0786666666667,
      work = 1674.2,
      activities = 230.4,
      places = 1661.5,
      bottle = 1.348,
      membership = 37.3406666666667
    ),
    "`scaled:scale`" = c(
      sunshine = 530.694790740912,
      pollution = 18.9123112126017,
      work = 186.337735641866,
      activities = 135.701396328546,
      places = 1547.98253019635,
      bottle = 0.645442377032191,
      membership = 11.818911477194
    )
  )

Answer 1

Some thoughts to hopefully help you... Do you have a reason to think that there should be clusters in the first place? If so, how many would you say is a sensible number?

In the end, clustering algorithms and selection methods will return some optimal number of clusters even if the data is mostly noise, unless it is really white noise, which is unusual in many real cases. In my experience, it is not surprising that different methods of clustering and assessment give different results since they rely on different assumptions.

I think it is also worth considering how you want to use the results. If you want to prove that there are some clusters, in my opinion the results you show are not convincing. However, if you want to use the clusters as a sort of data reduction to facilitate interpretation, then I wouldn't be too obsessed about finding the optimal number based on some metrics. I would instead think in terms of what number is more reasonable.

You could try also (h)dbscan which has the nice feature of allowing some datapoints to remain unclustered. But again, I wouldn't rely solely on algorithms to decide what is best.