1
$\begingroup$

I am working on a project for my university. A part of this project is to compare the influence of PCA on clustering. Therefore I have a football player dataset that contains a feature called "position group" which contains groups from 1 to 3. E.g. the heavy line players are in group 1, lighter receivers, cornerbacks etc are in group 2 and so on. Now I have to generate clusters with k-means and k-medoids based on 16 features that are fitness exercise results and body composition measurements like size and weight from each player. For this I use k = 3 because there are 3 player groups in the dataset. Goal of the clustering is to determine an "optimal theoretical" player allocation to a specific group so that I can say something like this: "3 Wide Receivers changed to the group of the heavy line men based on the clustering results. This could be an indication of a wrong position allocation from the coach. The coach should check this."
For every algorithm I use the same dataset with applied PCA and without applied PCA. That means I have 4 results in total.

Now I want to compare the results. I compare the clusters with the original data by using the rand index.
The methods do not differ a lot: 

Algorithm               Similarity to original clusters
K-means without PCA     0,514
K-means with PCA        0,544
K-medoids without PCA   0,528
K-medoids with PCA      0,532

Furhermore I use the intra- and inter-cluster similarity measures. The intra cluster distances are the following:

    Algorithm          Cluster 1    Cluster 2   Cluster 3
K-means without PCA    2,452        2,341       2,675
K-means with PCA       2,324        2,216       1,560
K-medoids without PCA  2,166        2,828       2,320
K-medoids with PCA     1,968        2,642       2,420

What is the best way to determine the best result especially for the intra cluster distances? Should I calculate the sum of each method and the smallest sum is the best approach?

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Unsupervised learning algorithms do not lend themselves to purely quantitative decisions. This is due to the fact that the choices involved in arriving at any solution are highly subjective and, therefore, are not optimizable. You need to provide a little more context to this question. For instance, would you describe the information (features) being clustered? Based on that, there would be good reasons for and against using PCA as a dimension reduction technique that would eliminate two of your results. Same goes for choosing between k-means and k-medoids. $\endgroup$
    – Mike Hunter
    Jan 25, 2016 at 10:45
  • $\begingroup$ How and why did you choose 3 clusters for each solution? Do these solutions make any intuitive sense? How well do they project to external or totally out-of-sample data? Why rely only on distance-based similarity measures? In other words, there is a whole lot of commonsense logic that can and should be leveraged before you make a purely quantitative choice between results. $\endgroup$
    – Mike Hunter
    Jan 25, 2016 at 10:45
  • $\begingroup$ Sorry for the little information. Please see my edited answer. $\endgroup$
    – Kewitschka
    Jan 25, 2016 at 11:12
  • 1
    $\begingroup$ Fair enough. Both algorithms are strict in requiring continuous metrics as input. Mixtures of scale types such as dummy (0,1) and other discrete features would require a different approach such as latent class clustering. Given that, do your 16 metrics adhere to this rule? Next, while k-means is generally robust to some outliers in the data, k-medoids is to be preferred when these outliers are large enough that the mean is biased. Are there outliers in your data? If so, how biased is the mean vs the median values? $\endgroup$
    – Mike Hunter
    Jan 25, 2016 at 12:33
  • 2
    $\begingroup$ Then there are the questions of evaluatory metrics. Given that you know which group these athletes have been assigned to, why not use a loss function, error or misclassification rate? The small sample size is only a barrier if you think of it as fixed. You can bootstrap the data, splitting it 50-50 into training vs validation across iterations and get an average error rate for each approach. There are plenty of other metrics to choose from, e.g., Marina Melia has an information theoretic metric that compares different cluster solutions stat.washington.edu/mmp/Papers/compare-colt.pdf $\endgroup$
    – Mike Hunter
    Jan 25, 2016 at 12:40

1 Answer 1

Reset to default
2
$\begingroup$

You must not expect clusters to agree with your known classes.

This may hold, but it does not necessarily hold.

It is a perfectly valid clustering result if it grouped your football players into three clusters that correspond to e.g.:

  • blonde hair
  • brunette hair
  • black hair

This is perfectly reasonable for an unsupervised method.

If you have a predefined task such as the three groups of players, then you should use a classifier instead.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Based on this document it is possible. It sais: " The gold standard is ideally produced by human judges with a good level of inter-judge agreement (...). We can then compute an external criterion that evaluates how well the clustering matches the gold standard classes." $\endgroup$
    – Kewitschka
    Jan 26, 2016 at 7:39
  • $\begingroup$ You can measure how well it agrees - but that does not mean it will always/often agree with the human judges. Just have the human judges put random labels there - how is the algorithm going to work? $\endgroup$
    – Has QUIT--Anony-Mousse
    Jan 26, 2016 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.