5
$\begingroup$

I have a k-means clustering result with 35 clusters, there are 5000 documents that each belong to one of the 35 cluster. I would like to visualize the results of the clustering algorithm on a scatter plot (or something similar) where each document is colored based on which cluster they belong to, and their distance on the visualization is proportional to their distance in similarity (i.e. the more similar they are, the closer they appear on the visualization). Ideally, it would also be nice to see the top 10 words that belong to the clusters. I am attaching my code for the clustering algorithm, it deals with data from a database.

myCorpus <- Corpus(VectorSource(userbios$bio))
    docs <- userbios$twitter_id
# convert to lower case
myCorpus <- tm_map(myCorpus, tolower)
# remove punctuation
myCorpus <- tm_map(myCorpus, removePunctuation)
# remove numbers
myCorpus <- tm_map(myCorpus, removeNumbers)
# remove URLs
removeURL <- function(x) gsub("http[[:alnum:]]*", "", x)
myCorpus <- tm_map(myCorpus, removeURL)
# add one extra stop words:  "via"
myStopwords <- c(stopwords('english'), "twitter", "tweets", "tweet", "tweeting", "account")


# remove stopwords from corpus
myCorpus <- tm_map(myCorpus, removeWords, myStopwords)


myTdm <- TermDocumentMatrix(myCorpus, control = list(wordLengths=c(1,Inf), weighting=weightTfIdf))
# remove sparse terms
myTdm2 <- removeSparseTerms(myTdm, sparse=0.90)

m2 <- as.matrix(myTdm2)
#cluster terms
distMatrix <- dist(scale(m2))
fit <- hclust(distMatrix, method="ward")
# transpose the matrix to cluster documents (tweets)
 m3 <- t(m2)

# k-means clustering
 k <- 35
kmeansResult <- kmeans(m3, k)
#cluster centers
round(kmeansResult$centers, digits=3)
    for (i in 1:k) {
      cat(paste("cluster ", i, ": ", sep=""))
      s <- sort(kmeansResult$centers[i,], decreasing=T)
      cat(names(s)[1:15], "\n")
      # print the tweets of every cluster + # 
      print(docs[which(kmeansResult$cluster==i)])
}
$\endgroup$
3
$\begingroup$

Try to analyze the differences of the clusters first.

K-means is an odd algorithm. Sometimes it just works very well, and in other situations it fails just very very badly. It has a tendency to just split your data set along a single axis.

So you may actually find out that your clustering result is something like this:

  • Cluster A contains all documents that contain "apple"
  • Cluster B contains all documents that contain "banana"
  • Cluster C contains all documents that contain "cocoa"
  • Cluster D contains all the others

You need to double check your clustering results!

There are several reasons for this. One is the way clusters look to k-means. They're Voronoi cells, separated by orthogonal hyperplanes. The other big effect here is the sparsity of your data set. The mean vectors computed by k-means will be much less sparse, usually. In fact, the average distance between the mean vectors will likely be lower than the distances from your data objects to the closest mean.

The latter probably is a good test: What is the average distance between to cluster centers, and what is the average distance of an observation to the nearest cluster center? Clearly, objects should on average be closer to their cluster center than two cluster centers to each other. But this may actually not hold for sparse data.

$\endgroup$
  • 1
    $\begingroup$ Thank you! Your last point is very interesting, it should be trivial, but I hadn't even thought about it. I will definitely try and check the distances between two cluster centers and an observation to the nearest cluster center! I validated the result of the clustering algorithm by printing out the top 15 words from each cluster and seeing if they made any sense, which is of course not a very precise way to do it. $\endgroup$ – d12n Feb 5 '13 at 17:57
2
$\begingroup$

When considering a scatter plot, what constitutes appropriate axes? The TF/IDF features used for clustering will not work.

Some ideas: - Build a graph, picking either clusters as nodes or documents as nodes, and distance as edges. - Multi-Dimensional scaling (MDS). http://en.wikipedia.org/wiki/Multidimensional_scaling

There are many other options you might consider, depending on what you are trying to accomplish with the visualization. For example, if you are trying to validate your results in some way, consider graphing cluster metrics such as cohesion and separation, or a silhouette plot.


You mentioned you'd like to look at the "top 10" words in each cluster. You could pick 10 words that are closest to the center of each cluster (representative points). However, if you are trying to use the visualization to see how successful your clustering was, this is a biased sample that will show you good cohesion and probably good separation between clusters.

$\endgroup$
  • $\begingroup$ Thank you for your answer, appropriate axes was my problem as well, I am very new to this, so I have no idea. I already have the top 10 words that I would like to use as my labels, I am just using the top 15 most frequent word in every cluster (I said 10 in the post, will change it now), it is the last for(1 in 1:k) that does that, but I just used that to see what words it turns into clusters, to check whether it makes sense. What I would like to accomplish with the visualization is to see at a glance the difference in numbers between clusters and how similar these clusters are to each other. $\endgroup$ – d12n Feb 5 '13 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.