# Visualizing the distribution of something within a very large body of data

Say I have a database of around a million words, and I want to get an intuitive idea about exactly how a particular, quite infrequent, word is distributed throughout this data. My goal is to be able to see clearly whether this word tends to cluster together, or whether it is relatively evenly spaced. What would be some good methods for visualizing this?

For instance, I have seen something that looks useful. It's basically a strip (long rectangle) in which each instance of something is represented by a very thin red vertical line. The problem is that I don't know what these are called, and therefore I can't figure out how to make something like this in R.

Any help finding the right R function for that, or any other suggestions for good ways to visualize this sort of data, would be most appreciated.

-
 Seems a little silly to post as an answer, but wordle may give you something useful, or at least interesting to ponder... – Chase Dec 8 '10 at 16:51

## 4 Answers

While Whuber is correct in principle you still might be able to see something because your word is very infrequent and you only want plots of the one word. Something quite uncommon might only appear 30 times, probably not more than 500. Let's say you convert your words into a single vector of words that's a million long. You could easily construct a plot with basic R commands. Let's call the vector 'words' and the rare item 'wretch'.

n <- length(words)
plot(1:n, integer(n), type = 'n', xlab = 'index of word', ylab = '', main = 'instances of wretch', yaxt = 'n', frame.plot = TRUE)
wretch <- which(words %in% 'wretch')
abline(v = wretch, col = 'red', lwd = 0.2)


You could change the line assigning wretch using a grep command if you need to account for variations of the word. Also, the lwd in the abline command could be set thicker or thinner depending on the frequency of the word. If you end up plotting 400 instances 0.2 will work fine.

I tried some density plots of this kind of data. I imported about 50,000 words of Shakespeare and finding patterns was easier for me in the code above than it was in the density plots. I used a very common word that appeared in frequency 200x more than the mean frequency ('to') and the plots looked just fine. I think you'll make a fine graph like this with rare instances in 1e6 words.

-

With a 1200 dpi printer using the thinnest possible line (one pixel) for each word, your plot of a million words would still be almost 20 meters long!

Maybe a density plot would be more helpful.

-
Hah, good point. Well, let's say I could break this down into about 400 different groups? – Alan H. Dec 8 '10 at 5:47
Sure, that might work. What you're doing then is tantamount to using the color value of a graphic to represent word density. Color values are difficult to interpret accurately; graphical representations that map the variable of interest to length or distance usually work best. That's exactly what a kernel density plot will do for you. By varying the half-width you can range from extreme detail to a gross overview. – whuber Dec 8 '10 at 5:55
haha nice one, whuber – suncoolsu Dec 8 '10 at 6:02

Let's take a derivative (difference) here, so instead of working with location, you work directly with what you want: distance.

Say word FOO appears in the text 30 times. Calculate the distance (number of other words) between each consecutive occurrence of FOO, creating a vector of 29 distances. Then pick your plot: histogram, density, xy with log x, etc.

This doesn't show you where clusters are, but it does show clustering.

-
You're on your way towards re-inventing Ripley's K function (stat.iastate.edu/preprint/articles/2001-18.pdf ). Note, however, that much information is lost in this approach. Suppose, for example, that 15 of the distances are zero and the other 15 equal 10,000. Is that 15 clusters of two words or one cluster of 16 words and 14 isolated words? – whuber Dec 9 '10 at 3:35
If you hadn't let me know, I might've named it Wayne's Method, written a paper on it, and gotten lots of references... or maybe that only works in medical literature... ;-) My moment of glory gone. On a more serious note, I guess for this exploratory application, the usual worrying about appropriate bandwidths isn't really an issue? – Wayne Dec 9 '10 at 16:58
(myself): just got back from lunch and realized that the usual bandwidth concerns are because you are using the kernel density to attempt to visualize an underlying population distribution from which you have sampled, while in this case the data is the entire population. Right? – Wayne Dec 9 '10 at 17:19

I don't know if this may be useful in your case, but in bioinformatics I often feel the need to visualize the distribution of gene counts in a give data set. This is definitely not as large as your data set, but I think the strategy can be followed for most of the large data sets.

A typical strategy would be to find a predetermined number of clusters using, say, hierarchical clustering (or any other clustering procedure). Once you have the clusters, you can sample a gene from each of these clusters. Assuming that the gene is representative of the cluster, visualizing the count for the gene (in form of density plot, histogram, qq-plot, etc.) is equivalent to visualizing the behavior of the cluster. You can do the same for all the clusters.

Basically, you reduce the huge data set to clusters and then visualize the representatives from these clusters assuming "on an average" the clusters' behavior will remain "more or less" the same.

Warning: This method is highly sensitive to a lot of things, a few are, clustering method, how many clusters you choose, etc.

I believe visualizing all the words if the number of words is reasonably large (say $\geq$ 50) would be pretty difficult. As as whuber aptly points out, it may be almost impossible.

-