0
$\begingroup$

I am having trouble understanding why Kmeans is returning so many unproportional clusters. For example, here are some of my test results in MATLAB after running my Kmeans algorithm on it: 

raw_crime_data = table2array(T(:,start_crime_stats_index:end_crime_stats_index))
k=15 % then I tried k=5
idxk = kmeans(raw_crime_data,k,'Distance','sqeuclidean');
for i=1:k
     length(unique(T.city(idxk(:) == i)))
end

k=15
    9442
    1
    2
    1
    3
    1
    5
    2738
    1
    6922
    2
    153
    8
    24
    3
k=5
    4299
    1
    5
    10191
    8

This issue just keeps happening.

Is it actually an issue?

Shouldn't I have proportional clusters?

Any pro tips on how to use Kmeans in such a way to group these data best?

It is just 10 completely numeric crime patterns.

I have also looked at this post, but it seems to be for text mining with Kmeans which is slightly different than clustering off of purely numeric data.

https://github.com/conradbm/data_science/blob/master/fbi_crime_1980_2014/data_manipulations.m

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Presumably, you have some outliers in your data. I would recommend that you take your k=5 data, find the point that was in a cluster by itself. Look at the distribution of distances from that point to all others. Look at the distribution of distances between all points. I would bet that they are very different distributions. $\endgroup$
    – G5W
    Commented Jan 17, 2017 at 22:46
  • $\begingroup$ This is true, the distributions are incredibly different. So eventually, without throwing out data, I just have to settle on a K that is good enough? Whats a good decision point on that? $\endgroup$
    – bmc
    Commented Jan 17, 2017 at 23:15
  • $\begingroup$ There is no clear answer to this. There is some guidance available. See This Cross Validated Post and This Stack Overflow Post. But ultimately, it is a choice you make depending on your problem. Both your k=5 and k=15 clusterings have a group with 8 points. Is that rubbish or a micro-cluster? You have to look at your data and your questions to decide. $\endgroup$
    – G5W
    Commented Jan 18, 2017 at 0:17

1 Answer 1

Reset to default
1
$\begingroup$

I had a similar issue on another platform (PySpark), and I think it's independent of the coding because it's a data issue. Try reframing the data either with logical quantiles, or by removing high-leverage points (One approach using linear models).

The concept behind this - if your data are crime rates by city, a city with a very high auto theft rate may form its own cluster, and the KMeans algorithm, in trying to maximize distance between clusters while minimizing distance between same-cluster points, will be reluctant to split up a much larger cluster with less variation. The high-leverage points can be pushed into their own cluster, allowing the algorithm to try to split up the much larger cluster.

I've personally seen Quantiling take a KMeans solution in which one of the factors dominates and turn it into a solution in which all factors are valid. The quantiles artificially standardize variation by reducing the number of possible values that a cluster can be fit upon. Do you know which of your features has the highest variance or seems to be driving most of the cluster solutions?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.