1
$\begingroup$

In my data I have 3 clusters with average silhouette 0.61 and very few negative values. I repeated k-means 10 times with k ranging from 2 to 10. This seemed to work ok, but the problem is that I got those results without scaling from 0 to 1, which I read is a required preprocessing before applying k-means.

Another potential problem is that my correlation matrix is relatively weak compared with iris data set (look at pairs graphics & correlation matrix). Is this a problem for k-means? With all the attributes I tried I got a pretty concentrated pairs graph. Removing outliers and scaling didn't help a lot.

Can I get quality clusters with that kind of data? (first slide1). Do I need to process the data with some other technique first? Or is it that my data are not suitable for applying k-means or other clustering method?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ I've rewritten your question a bit to try to make it a bit clearer what your questions are. I hope I haven't changed the sense. Just re-edit if I have. $\endgroup$
    – conjugateprior
    Commented Feb 1, 2013 at 11:23

1 Answer 1

Reset to default
2
$\begingroup$

k-means is a pretty dumb method. It minimizes variances in the assignment, no matter whether this is intuitive or not. If you have a uniform distributed data set and run k-means with k=3, you will get three clusters. If you run with k=4, you will get four clusters, and the sum of variances will be lower (i.e. the clustering will be "better").

The problem is that for a lot of users, sum-of-squares is not a good quality indicator.

Whenever you are using clustering, you should visualize the produced clusters and double-check them whether they make sense for you personally, not just for some mathematical statistic. It is not as if the mathematical statistic can objectively measure your intuition.

“clusters are in the eye of the beholder”

is a pretty smart statement. What is a good cluster and what is not is 99% an application- and user specific thing, and nothing you could measure objectively by sum-of-squares etc.

Improve this answer
$\endgroup$
9
  • $\begingroup$ After scaling (0-1) dataset from the first slide, I visualize clusters, so I will be grateful to look at the slides and give me a comment. I know that I need to know if clusters make sense from attributes aspects, but I need data miner technical opinion about results. Do you will consider that your job is done if you have an order from your customer, on that specific data, with results as you can see on slides (docs.google.com/presentation/d/…)? $\endgroup$
    – Mario Župan
    Commented Feb 3, 2013 at 8:22
  • $\begingroup$ Please comment every slide to let me know if I'm on a right way and to let me know what I need to learn in cluster analysis. Problem is that I always learn techniques on iris data which are already clustered by attributes values. $\endgroup$
    – Mario Župan
    Commented Feb 3, 2013 at 8:26
  • $\begingroup$ Just running some algorithms and showing the results does not help much. You could probably generate these fully automatically. You need to think more about the results, about what you can learn from them. Slides 12+13 for example shows that your clustering is 1 dimensional, it's trivial slices on the attribute X4Solvency. So I'd say, the result is meaningless, sorry. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Feb 3, 2013 at 11:47
  • $\begingroup$ I see, then you leads me to conclusion that my attributes choice is not significant for my financial performance analysis. I need to change attributes, right? But what do you think about procedure of cluster analysis, according to slides? Do I need another technique or meausure for my pilot project? PROJECT TARGET: cluster subjects according to their financial performances $\endgroup$
    – Mario Župan
    Commented Feb 3, 2013 at 18:20
  • $\begingroup$ Procedure: 1. choosing attributes (I plan to use correlation matrix ie 1th slide visualization). What procedure you suggest for choosing attributes, significant for kmeans? 2. correction of distribution (removing outliers, logarithm and scale from 0 to 1). 3. kmeans iterations for choosing this one with smallest sumOfSquare within and largest between clusters 4. observing clusters according to a silhouette, pairs and other graphs as slides shows. $\endgroup$
    – Mario Župan
    Commented Feb 3, 2013 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.