R: Visualizing document clustering results

Question

I have a k-means clustering result with 35 clusters, there are 5000 documents that each belong to one of the 35 cluster. I would like to visualize the results of the clustering algorithm on a scatter plot (or something similar) where each document is colored based on which cluster they belong to, and their distance on the visualization is proportional to their distance in similarity (i.e. the more similar they are, the closer they appear on the visualization). Ideally, it would also be nice to see the top 10 words that belong to the clusters. I am attaching my code for the clustering algorithm, it deals with data from a database.

myCorpus <- Corpus(VectorSource(userbios$bio))
    docs <- userbios$twitter_id
# convert to lower case
myCorpus <- tm_map(myCorpus, tolower)
# remove punctuation
myCorpus <- tm_map(myCorpus, removePunctuation)
# remove numbers
myCorpus <- tm_map(myCorpus, removeNumbers)
# remove URLs
removeURL <- function(x) gsub("http[[:alnum:]]*", "", x)
myCorpus <- tm_map(myCorpus, removeURL)
# add one extra stop words:  "via"
myStopwords <- c(stopwords('english'), "twitter", "tweets", "tweet", "tweeting", "account")


# remove stopwords from corpus
myCorpus <- tm_map(myCorpus, removeWords, myStopwords)


myTdm <- TermDocumentMatrix(myCorpus, control = list(wordLengths=c(1,Inf), weighting=weightTfIdf))
# remove sparse terms
myTdm2 <- removeSparseTerms(myTdm, sparse=0.90)

m2 <- as.matrix(myTdm2)
#cluster terms
distMatrix <- dist(scale(m2))
fit <- hclust(distMatrix, method="ward")
# transpose the matrix to cluster documents (tweets)
 m3 <- t(m2)

# k-means clustering
 k <- 35
kmeansResult <- kmeans(m3, k)
#cluster centers
round(kmeansResult$centers, digits=3)
    for (i in 1:k) {
      cat(paste("cluster ", i, ": ", sep=""))
      s <- sort(kmeansResult$centers[i,], decreasing=T)
      cat(names(s)[1:15], "\n")
      # print the tweets of every cluster + # 
      print(docs[which(kmeansResult$cluster==i)])
}

Answer 1

Try to analyze the differences of the clusters first.

K-means is an odd algorithm. Sometimes it just works very well, and in other situations it fails just very very badly. It has a tendency to just split your data set along a single axis.

So you may actually find out that your clustering result is something like this:

• Cluster A contains all documents that contain "apple"
• Cluster B contains all documents that contain "banana"
• Cluster C contains all documents that contain "cocoa"
• Cluster D contains all the others

You need to double check your clustering results!

There are several reasons for this. One is the way clusters look to k-means. They're Voronoi cells, separated by orthogonal hyperplanes. The other big effect here is the sparsity of your data set. The mean vectors computed by k-means will be much less sparse, usually. In fact, the average distance between the mean vectors will likely be lower than the distances from your data objects to the closest mean.

The latter probably is a good test: What is the average distance between to cluster centers, and what is the average distance of an observation to the nearest cluster center? Clearly, objects should on average be closer to their cluster center than two cluster centers to each other. But this may actually not hold for sparse data.

Answer 2

When considering a scatter plot, what constitutes appropriate axes? The TF/IDF features used for clustering will not work.

Some ideas: - Build a graph, picking either clusters as nodes or documents as nodes, and distance as edges. - Multi-Dimensional scaling (MDS). http://en.wikipedia.org/wiki/Multidimensional_scaling

There are many other options you might consider, depending on what you are trying to accomplish with the visualization. For example, if you are trying to validate your results in some way, consider graphing cluster metrics such as cohesion and separation, or a silhouette plot.

---

You mentioned you'd like to look at the "top 10" words in each cluster. You could pick 10 words that are closest to the center of each cluster (representative points). However, if you are trying to use the visualization to see how successful your clustering was, this is a biased sample that will show you good cohesion and probably good separation between clusters.