0
$\begingroup$

I am having trouble understanding why Kmeans is returning so many unproportional clusters. For example, here are some of my test results in MATLAB after running my Kmeans algorithm on it:

raw_crime_data = table2array(T(:,start_crime_stats_index:end_crime_stats_index))
k=15 % then I tried k=5
idxk = kmeans(raw_crime_data,k,'Distance','sqeuclidean');
for i=1:k
     length(unique(T.city(idxk(:) == i)))
end

k=15
    9442
    1
    2
    1
    3
    1
    5
    2738
    1
    6922
    2
    153
    8
    24
    3
k=5
    4299
    1
    5
    10191
    8

This issue just keeps happening.

Is it actually an issue?

Shouldn't I have proportional clusters?

Any pro tips on how to use Kmeans in such a way to group these data best?

It is just 10 completely numeric crime patterns.

I have also looked at this post, but it seems to be for text mining with Kmeans which is slightly different than clustering off of purely numeric data.

https://github.com/conradbm/data_science/blob/master/fbi_crime_1980_2014/data_manipulations.m

$\endgroup$
3
  • 1
    $\begingroup$ Presumably, you have some outliers in your data. I would recommend that you take your k=5 data, find the point that was in a cluster by itself. Look at the distribution of distances from that point to all others. Look at the distribution of distances between all points. I would bet that they are very different distributions. $\endgroup$
    – G5W
    Jan 17 '17 at 22:46
  • $\begingroup$ This is true, the distributions are incredibly different. So eventually, without throwing out data, I just have to settle on a K that is good enough? Whats a good decision point on that? $\endgroup$
    – bmc
    Jan 17 '17 at 23:15
  • $\begingroup$ There is no clear answer to this. There is some guidance available. See This Cross Validated Post and This Stack Overflow Post. But ultimately, it is a choice you make depending on your problem. Both your k=5 and k=15 clusterings have a group with 8 points. Is that rubbish or a micro-cluster? You have to look at your data and your questions to decide. $\endgroup$
    – G5W
    Jan 18 '17 at 0:17
1
$\begingroup$

I had a similar issue on another platform (PySpark), and I think it's independent of the coding because it's a data issue. Try reframing the data either with logical quantiles, or by removing high-leverage points (One approach using linear models).

The concept behind this - if your data are crime rates by city, a city with a very high auto theft rate may form its own cluster, and the KMeans algorithm, in trying to maximize distance between clusters while minimizing distance between same-cluster points, will be reluctant to split up a much larger cluster with less variation. The high-leverage points can be pushed into their own cluster, allowing the algorithm to try to split up the much larger cluster.

I've personally seen Quantiling take a KMeans solution in which one of the factors dominates and turn it into a solution in which all factors are valid. The quantiles artificially standardize variation by reducing the number of possible values that a cluster can be fit upon. Do you know which of your features has the highest variance or seems to be driving most of the cluster solutions?

$\endgroup$

This site is temporarily in read only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .