# How to visualize an enormous sparse contingency table?

I have two variables: Drug Name (DN) and corresponding Adverse Events (AE), which stand in a many-to-many relation. There are 33,556 drug names and 9,516 adverse events. The sample size is about 5.8 million observations.

I want to study and understand the association/relationship between DN and AE. I’m thinking about a way of visualizing this set in R since it's better to look at pictures. I'm not sure how to go about it...

-
(+1) In light of the additional information presented in these comments (since deleted, because the information now appears in the question itself), this has become an interesting and challenging problem. I encourage previous downvoters to change their votes in recognition of this (and others to upvote the question if you agree with me!). –  whuber Jul 19 '12 at 14:26
Are the data publicly available? –  cardinal Jul 19 '12 at 14:30
@cardinal, yes.. the data obtained from FDA, it's called AERS. See the link: fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/… –  act00 Jul 19 '12 at 16:40
+1 for all prompt updates and linking to publicly available data. I hope you'll get some good response here. –  chl Jul 19 '12 at 21:59
add comment

## 2 Answers

What you could do is use the residual shading ideas from vcd here in combination with sparse matrix visualisation as for example on page 49 of this book chapter. Imagine the latter plot with residual shadings and you get the idea.

The sparse matrix/contigency table would normally contain the number of occurences of each drug with each adverse effect. With the residual shading idea however you can set up a baseline log linear model (e.g. an independence model or whatever else you like) and use the color scheme to find out which drugs/effect combination occurs more often/ less often than the model would predict. Since you have many observations, you could use a very fine color thresholding and get a map that looks similar to how microarrays in cluster analysis are often visualised e.g. here (but probably with stronger color "gradients"). Or you could build the thresholds such that only if the differences of observations to predictions exceeds the threshold than it gets colored and the rest will remain white. How exactly you would do this (e.g. which model to use or which thresholds) depends on your questions.

Edit So here's how I would do it (given I'd have enough RAM available...)

1. Create a sparse matrix of the desired dimensions (drug names x effects)
2. Calculate the residuals from the independence loglinear model
3. Use a color gradient in fine resolution from the min to the maximum of the residual (e.g. with a hsv colorspace)
4. Insert the according color value of the residuals magnitude at the according position in the sparse matrix
5. Plot the matrix with an image plot.

You then end up with something like this (of course you picture will be much larger and there will be a much lower pixel size but you should get the idea. With clever usage of color you can visualize the associations/departures from independence you are most interested in).

A quick and dirty example with a 100x100 matrix. This is just a toy example with residuals ranging from -10 to 10 as you can see in the legend. White is zero, blue is less frequent than expected, red is more frequent than expected. You should be able to get the idea and take it from there. Edit: I fixed the plot's set up and used non-violent colors.

This was done using the image function and cm.colors() in the following function:

ImagePlot <- function(x, ...){
min <- min(x)
max <- max(x)
layout(matrix(data=c(1,2), nrow=1, ncol=2), widths=c(1,7), heights=c(1,1))

ColorLevels <- cm.colors(255)

# Color Scale
par(mar = c(1,2.2,1,1))
image(1, seq(min,max,length=255),
matrix(data=seq(min,max,length=255), ncol=length(ColorLevels),nrow=1),
col=ColorLevels,
xlab="",ylab="",
xaxt="n")

# Data Map
par(mar = c(0.5,1,1,1))
image(1:dim(x)[1], 1:dim(x)[2], t(x), col=ColorLevels, xlab="",
ylab="", axes=FALSE, zlim=c(min,max))

layout(1)
}

#100x100 example
x <- c(seq(-10,10,length=255),rep(0,600))
mat <- matrix(sample(x,10000,replace=TRUE),nrow=100,ncol=100)
ImagePlot(mat)


using ideas from here http://www.phaget4.org/R/image_matrix.html. If your matrix is so big that the image function gets slow, use the useRaster=TRUE argument (you might also want to use sparse Matrix objects; note that there should be an image method if you want to use the code from above, see the sparseM package.)

If you do this, some clever ordering of the rows/columns might become handy, which you can calculate with the arules package (check page 17 and 18 or so). I would generally recommend the arules utilities for this type of data and problem (not only visualisation but also to find patterns). There you will also find measures of association between the levels that you could use instead of the residual shading.

You might also want to look at tableplots of you want to investigate only a couple of adverse effects later.

-
Many thanks!!! This is very helpful! –  act00 Jul 23 '12 at 2:31
add comment

This is a known problem with a large literature: You could have a look at, for example (from statistics in medicine, downloadable): "Bayesian pharmacovigilance signal detection methods revisited in a multiple comparison setting " and references therin.

-
It sounds like an interesting paper (not sure it is freely available, but there's an R package). Now, what graphical solution would you suggest? –  chl Jul 22 '12 at 10:03
That paper is freely downloadable! As for graphical solution, your table is clearly to large for heatmap-type solution being viable. Maybe, first do some model, like that above, Then reorder rows/cols accordin to some measute of "interest" calculated from the model. Then, just maybe, a heatmap of the most inbteresting part could do! –  kjetil b halvorsen Jul 23 '12 at 21:12
add comment