I have a general doubt regarding clustering. I have a data set of size 1196*18675. where 1196 is the no of documents. I am trying to cluster the data with k=7 using k-means. Each time the clustered group varies in size.

[iter] c1 c2 c3 c4 c5 c6 c7

[500] 346 233 151 128 126 125 87

[1000] 286 162 149 146 166 99 188

How can I judge this result.Is their any method available to evaluate without the prior knowledge of each class documents.


How do you know that 7 is the number of clusters?

In any case, I suppose you are after what is called cluster validation.

The basic idea is that you compare inner-cluster distances vs. cross-cluster distances. If your clustering is good then you expect small distances between samples within the cluster (for some metric of distance), and large distances between clusters.

  • $\begingroup$ In this scenario I have a prior knowledge for choosing the value K as 7. $\endgroup$
    – sam
    Jul 16 '14 at 5:31
  • 2
    $\begingroup$ To add to @iliasfl's good answer, clustering, unlike, e.g., OLS regression, has no single "right" solution, so some variation in cluster membership is to be expected on different trials. $\endgroup$
    – rolando2
    Jul 16 '14 at 10:19

Also note that the k-means algorithm suffers from what is called the Curse of Dimensionality. This is where the more dimensions the data has (the 18675 in your case), the more unreliable the results of k-means is.

There are algorithms which perform better with higher dimensions, which you should look into. Alternatively, look into Principal Component Analysis, or the kernelized version of it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.