1
$\begingroup$

Consider the following plot.

I want to identify the regions that are considerably higher than the highest cluster. (The obvious regions which should be identified as their own clusters, notably at the x coordinate ~10 e+07. How would I be able to identify that using a clustering algorithm?

I am using R algorithm kmeans in the picture above. with 6 centers:

kmeans(numbers_vector, centers=6, nstart=10)

What can I do to alleviate the inadequacy of this algorithm? Use a different clustering algorithm? Have more centers? But If I have more centers, it identifies many more regions in the center (namely clusters 3,5, and 6). Any ideas?

Here is histogram and density. It is important to note that the histogram and density plot DO NOT show the spike ~x=10e+07, because the number of points involved in the spike ~20, perhaps are completely overshadowed by the ~107,350 points plotted.

enter image description here

enter image description here

Data: x (88289 obs.); Bandwidth 'bw' = 0.7574

   x                 y

Min. : -1.272 Min. :0.000e+00
1st Qu.: 40.114 1st Qu.:4.110e-06
Median : 81.500 Median :3.167e-05
Mean : 81.500 Mean :6.035e-03
3rd Qu.:122.886 3rd Qu.:2.886e-03
Max. :164.272 Max. :4.827e-02

Improve this question
$\endgroup$
8
  • $\begingroup$ Are these questions ever answered on stats.stackexchange.com, or will they just die if no one answers within an allotted period? Feel like SO provides a much faster response. $\endgroup$
    – user41912
    Jul 28, 2014 at 20:43
  • 1
    $\begingroup$ Isn't fifteen minutes a little soon to declare the question dead? $\endgroup$
    – David Robinson
    Jul 28, 2014 at 21:11
  • $\begingroup$ In any case: this is not a good case for k-means clustering. K-means clustering is generally designed for when you have multiple clusters that are each approximately normally distributed (or multivariate normal) around a different center. Your y-axis points don't look like they fall into several clusters: rather, I would guess that they follow a very roughly normal distribution around about 60, with some outliers. This will be the key to looking for values that are outliers. Could you show a histogram of your numbers_vector? $\endgroup$
    – David Robinson
    Jul 28, 2014 at 21:14
  • $\begingroup$ @David Robinson Yep, uploading it in 2 minutes after I generate in R. If you want the dataset, look at the comments I left to the answer provided to the question by Eric. $\endgroup$
    – user41912
    Jul 28, 2014 at 21:24
  • 1
    $\begingroup$ First, rather than looking and quantiles I would suggest considering the problem parametrically. The data looks rather close to normal with the exception of a group of outliers at 0. First remove those 0s, then find the mean and standard deviation. For choosing the number of sd's away from the mean that are "significant": you could use that distribution to compute a p-value for each point (probability you would see that extreme or more if it were actually normal). Then it would become a problem of multiple hypothesis testing to find which points are actually extreme given how many you have $\endgroup$
    – David Robinson
    Jul 29, 2014 at 3:28

2 Answers 2

Reset to default
1
$\begingroup$

K-means will partition to minimize variance.

The minimum variance parition does produce these slices.

Instead of using clustering, use density estimation. You have already plotted densities for your data. It's fairly obvious how to identify low density regions in the histogram (if you don't want to use bins, you can use kernel density estimation). The low values will also be separated from the majority of the data by a low density region.

The spike you are interested in is an area with really low density.

I.e. split your data set at a density minimum, and select data in areas with low density.

Improve this answer
$\endgroup$
0
$\begingroup$

How would I be able to identify that using a clustering algorithm?

Assuming you want to identity outliers or where the groups break at (your question isn't exactly clear to me), I recommend plotting your data and then plotting your centers. This will plot your data points color coded by cluster and then plot the cluster centers over it.

c1<-kmeans(numbers_vector, centers=6, nstart=10)
plot(numbers_vector, col = cl1$cluster)
    points(cl1$centers, col = 11, pch = 8, cex = 2,lwd=5)

Without actual data, I can't be much more help.

I am using R algorithm kmeans in the picture above. with 6 centers:

The picture above appears not to have shown through. Can you provide your dataset?

What can I do to alleviate the inadequacy of this algorithm? Use a different clustering algorithm? Have more centers? But If I have more centers, it identifies many more regions in the center (namely clusters 3,5, and 6). Any ideas?

Again, without data your questions are ambiguous and difficult to answer.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This is the link to the picture. i.stack.imgur.com/QbZ5T.png $\endgroup$
    – user41912
    Jul 28, 2014 at 21:18
  • $\begingroup$ The dataset I plan to upload to dropbox shortly as an .RData file. Simply download the file from dropbox and when you open R, use the command load(file_location), and then use head(df.add) to confirm the loading of the dataset. The x values are the start column and the y values are the Number_of_AID_OH column. Ignore all other columns. $\endgroup$
    – user41912
    Jul 28, 2014 at 21:21
  • $\begingroup$ dropbox.com/s/h9tkh8x9u1dqy8z/clustering.RData. This is the link here. $\endgroup$
    – user41912
    Jul 28, 2014 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.