Visualizing the distribution of something within a very large body of data

Question

Say I have a database of around a million words, and I want to get an intuitive idea about exactly how a particular, quite infrequent, word is distributed throughout this data. My goal is to be able to see clearly whether this word tends to cluster together, or whether it is relatively evenly spaced. What would be some good methods for visualizing this?

For instance, I have seen something that looks useful. It's basically a strip (long rectangle) in which each instance of something is represented by a very thin red vertical line. The problem is that I don't know what these are called, and therefore I can't figure out how to make something like this in R.

Any help finding the right R function for that, or any other suggestions for good ways to visualize this sort of data, would be most appreciated.

Answer 1

While Whuber is correct in principle you still might be able to see something because your word is very infrequent and you only want plots of the one word. Something quite uncommon might only appear 30 times, probably not more than 500. Let's say you convert your words into a single vector of words that's a million long. You could easily construct a plot with basic R commands. Let's call the vector 'words' and the rare item 'wretch'.

n <- length(words)
plot(1:n, integer(n), type = 'n', xlab = 'index of word', ylab = '', main = 'instances of wretch', yaxt = 'n', frame.plot = TRUE)
wretch <- which(words %in% 'wretch')
abline(v = wretch, col = 'red', lwd = 0.2)

You could change the line assigning wretch using a grep command if you need to account for variations of the word. Also, the lwd in the abline command could be set thicker or thinner depending on the frequency of the word. If you end up plotting 400 instances 0.2 will work fine.

I tried some density plots of this kind of data. I imported about 50,000 words of Shakespeare and finding patterns was easier for me in the code above than it was in the density plots. I used a very common word that appeared in frequency 200x more than the mean frequency ('to') and the plots looked just fine. I think you'll make a fine graph like this with rare instances in 1e6 words.

Answer 2

With a 1200 dpi printer using the thinnest possible line (one pixel) for each word, your plot of a million words would still be almost 20 meters long!

Maybe a density plot would be more helpful.

Answer 3

Let's take a derivative (difference) here, so instead of working with location, you work directly with what you want: distance.

Say word FOO appears in the text 30 times. Calculate the distance (number of other words) between each consecutive occurrence of FOO, creating a vector of 29 distances. Then pick your plot: histogram, density, xy with log x, etc.

This doesn't show you where clusters are, but it does show clustering.

Answer 4

I don't know if this may be useful in your case, but in bioinformatics I often feel the need to visualize the distribution of gene counts in a give data set. This is definitely not as large as your data set, but I think the strategy can be followed for most of the large data sets.

A typical strategy would be to find a predetermined number of clusters using, say, hierarchical clustering (or any other clustering procedure). Once you have the clusters, you can sample a gene from each of these clusters. Assuming that the gene is representative of the cluster, visualizing the count for the gene (in form of density plot, histogram, qq-plot, etc.) is equivalent to visualizing the behavior of the cluster. You can do the same for all the clusters.

Basically, you reduce the huge data set to clusters and then visualize the representatives from these clusters assuming "on an average" the clusters' behavior will remain "more or less" the same.

Warning: This method is highly sensitive to a lot of things, a few are, clustering method, how many clusters you choose, etc.

I believe visualizing all the words if the number of words is reasonably large (say $\geq$ 50) would be pretty difficult. As as whuber aptly points out, it may be almost impossible.