Cluster Analysis: effectiveness of k means results and alternative methods

I have to separate 425 observations based on certain variables numbering 32.

1)I used PCA to reduce the dimensionality of Data, which gave me 32 components out of which 5 components accounted for 75% of the variance.

2)I used Elbow plot to determine no of clusters that is 6. I further used these 5 components as variables for kmeans clustering. when plotted using clusplot() Clusters are not well separated.

Does this mean that clustering was not successful.(between_SS / total_SS = 67.9 %). Is there any other way to determine effective ness of clustering? Can somebody suggest a better way to obtain better clusters may be PAM, Hierarchical clustering ...?

• Notice that your choice of the number of cluster does not seems obvious for me according to the elbow graph. If you do this on the fly, why not considering 7 clusters? There seems to be a small drop around 10 and stabilizing as well. For each of these values you can calculate between_SS/total_SS – Colonel Beauvel Apr 28 '15 at 7:49
• Don't do factor-cluster analysis: ro.uow.edu.au/cgi/…. – tchakravarty Apr 28 '15 at 8:27
• There is no elbow in the elbow plot. Either there are no clusters, or you did not preprocess the data well. – Has QUIT--Anony-Mousse Apr 28 '15 at 23:00

First, let me state that I'm highly experienced in building cluster solutions in applied research contexts. Therefore, I have a strong, biased POV that does not agree with the objections to factor-cluster approaches raised in the pdf cited by Fg Nu. In other words, DO use factor-cluster methods in developing a cluster solution.

Among the reasons cluster analysis is considered "soft" and a-scientific by many has to do with the plethora of subjective decisions that are required in generating solutions. These include ground truth considerations such as the selection of inputs, whether one does a PCA or CFA, orthogonal vs oblique solutions and rotations to simple structure, not to mention the blizzard of arbitrary heuristics available for selecting the number of factors, the number of clusters, the impossibility of developing unique and optimizable solutions, and so on. My point is that adopting or acting on the recommendations in this pdf will not make any resulting solution more rigorous or "scientific."

The objections to factor-cluster analysis in the pdf cited by Fg Nu are primarily academic, red herring concerns that can be applied to any and all dimension reducing techniques. The irony is that the authors of this pdf are marketers writing about consumer segmentations in tourism, an applied context. That said, I am agnostic about whether or not their observations might have relevance in more truly "scientific" contexts such as writing a PhD dissertation, in cladistics and speciation -- the assignment of species to groups -- or in machine learning situations leveraging automatic approaches to k-means, etc.

Given that, the pdf's specific concern with whether or not the resulting solution is evaluated or profiled based on the factors from the transformed space or with the original variables is a purely academic issue. Transformations of any nature exact a price, whether it be in reshaping the latent geometric space as in mean standardization, in rescaling, whatever. Being cognizant of the tradeoffs is the key, not tossing an approach out because there are "problems." In an applied context where understanding and insight is the focus, this is an irrelevant concern.

Next, their observation about "throwing away" 40% of the original variance in using FA is true but so what? It's an artificial loss or number, in the first place. In the next place, motivate the logic behind why I should care. That wasn't done, at least not to my satisfaction.

Finally, objection #3 that factor-cluster solutions perform "worse" relative to the "correct" structure on known, artificial data is the biggest red herring of all. The fact is that the "correct structure" of data in real world analysis is never known. Next, the point of cluster analysis in applied contexts is not precision, its directional, strategic insight that is riddled with error.

Cluster solutions are well-known to be highly sensitive to redundancy. The strongest argument in favor of factor-cluster approaches is that factorization reduces or even removes redundancy in the inputs. This is the real beauty of factorization -- the fact that the resulting solution absorbs information about all of the inputs, making the selection of specific variables moot. This is not accomplished without a price, of course. Next, variable selection is a highly subjective, arbitrary step about which every statistician and their brother has a working paper and solution. Arriving at agreement on the "correct," raw inputs to use in the absence of factorization can and has derailed many clustering projects.

Ok...enough of my polemic. I want to note that, based on the scatterplot of dim1 by dim2, the solution looks pretty crisp and good to me.

One way to cross-validate the results would be to split your data into "train and test" sets, calibrate a solution on the training set, score the test data with that solution and then see how well you can recover the test solution based on a misclassification rate using something like a discriminant analysis based on LOOCV or jacknifing. One can do this CV in two ways: first, based on the "full" set of variables used in developing the factors/clusters. And, second, leveraging a "reduced" set of information that would be comparable to the poor quality and incomplete information available from huge data vendors such as Experian. The resulting misclassification rates should generate intuition as well as confidence (or not) about how well the cluster solutions are performing once they are projected and used based on the scoring of external information.

• Pedantically, or otherwise, speciation is what the organisms do; it is not the part of taxonomy whereby biologists distinguish species. More importantly, you almost make me want to read the paper you criticise to see how bad it really is.... Please don't use "academic" as a catch-all; if you want to say silly, wrong, confused, irrelevant, .... and explain why you say so, then that is naturally fine. Using "academic" abusively is too easy, and FWIW does not appeal to academics. – Nick Cox Oct 30 '15 at 12:53
• Speciation is not pedantic since it's evolutionary. Classification is or can be pedantic. I think that's my point. And you are correct about my use of academic. I will monitor that in future. – Mike Hunter Oct 30 '15 at 13:12
• "Pedantic" was a reference to the correction that followed it: I am pointing out that speciation is not a kind of classification. No more. – Nick Cox Oct 30 '15 at 13:41
• I feel that you might want to open a special thread about cluster analysis after dimensionality reduction (PCA, FA) - its pros and cons. There you could present your arguments as an answer to own question. And people could add their answers. As for currently, your (quite interesting) account is more about the article once mentioned in a comment, not so much about the OP's question. – ttnphns Mar 9 '16 at 8:09
• @ttnphns My post is a bit of a diatribe that I had completely forgotten about. Thanks for the suggestion. I will give it some thought. – Mike Hunter Mar 9 '16 at 12:02

It was described here: http://www.sciencemag.org/content/344/6191/1492.short

and has been implemented in R: https://github.com/thomasp85

• The big miracle is: how did they get THAT accepted?!? I guess they don't have any clustering experts in the reviewers board... kind of like the MIC coefficient which was on Nature or Science, and all the statistics community went WTF? – Has QUIT--Anony-Mousse May 1 '15 at 10:17
• @Anony-Mousse Have not read the Science paper -- care to elaborate? – tchakravarty May 3 '15 at 5:29
• It's ignoring most of the state of the art. – Has QUIT--Anony-Mousse May 3 '15 at 15:32
• @Anony-Mousse I'd actually find it extremely valuable if you took the time to explain (perhaps with reference to specific concepts used or not used in the paper itself) (as I think @T C asks) . I definitely am very under-educated on this and I would love to learn (for eg. it's not immediately obvious to me what one should criticize or review when reading literature of the sort). Thanks ahead of time. – npjc May 3 '15 at 20:07