A good way to show lots of data graphically

I'm working on a project that involves 14 variables and 345,000 observations for housing data (things like year built, square footage, price sold, county of residence, etc). I'm concerned with trying to find good graphical techniques and R libraries that contain nice plotting techniques.

I'm already seeing what in ggplot and lattice will work nicely, and I'm thinking of doing violin plots for some of my numerical variables.

What other packages would people recommend for displaying a large amount of either numerical or factor-typed variables in a clear, polished, and most importantly, succinct manner?

-
"clear, polished, and most importantly, succinct manner" Sounds like ggplot2 to me. – Brandon Bertelsen Sep 30 '10 at 4:49
I'm looking not just for R libraries, but also any specific types of graphs. My knowledge of graphs is limited to scatter, box, qq, histograms, violins, kernel density estimations, etc. Any slightly more obscure graphs that can reveal more about the data than those would be fantastic. – Christopher Aden Sep 30 '10 at 4:52
parallel coordinates sounds like another one to mention. Also methods of dimensionality reduction might be helpful. – Tal Galili Sep 30 '10 at 8:09

The best "graph" is so obvious nobody has mentioned it yet: make maps. Housing data depend fundamentally on spatial location (according to the old saw about real estate), so the very first thing to be done is to make a clear detailed map of each variable. To do this well with a third of a million points really requires an industrial-strength GIS, which can make short work of the process. After that it makes sense to go on and make probability plots and boxplots to explore univariate distributions, and to plot scatterplot matrices and wandering schematic boxplots, etc, to explore dependencies--but the maps will immediately suggest what to explore, how to model the data relationships, and how to break up the data geographically into meaningful subsets.

-
Great idea! I have latitudes and longitudes of all datapoints already, so such a task would be relatively elementary. I was thinking the maps library would be a good way to go, unless there's something better. – Christopher Aden Sep 30 '10 at 20:43
@Christopher You can also do this with ggplot2 (esp. if you don't need to draw country boundaries), had.co.nz/ggplot2/coord_map.html. Otherwise, maps, gmaps are better. There's also GeoXp and an R interface to GRASS. BTW, Mondrian has a plugin for geographical data :) – chl Sep 30 '10 at 21:12
Assigning a best answer can be difficult when there's several great suggestions, but I feel this is the right direction, keeping "succinct" in mind. I will give ggplot2 a try, and take a look at maps, GeoXp, and Mondrian. Thanks for the idea of graphing spatially! – Christopher Aden Oct 1 '10 at 17:34
I've had good success with lattice's levelplot and contourplot. Package fields has some nice features, including quiltplot which is nice if your data is not strictly on a grid. It also has a nice thin-plate spline function Tps to smooth non-gridded data to a grid. As for dedicated GIS software, GRASS somehow never makes sense to me, I prefer QGIS. – Wayne Sep 2 '12 at 13:37

I feel you are actually asking two questions: 1) what types of visualizations to use and 2) what R package can produce them.

In the case of what type of graph to use, there are many, and it depends on your needs (e.g: types of variables - numeric, factor, geographic etc, and the type of connections you are interested to display):

• If you have many numeric variables, you might want to use a scatter plot matrix (have a look here)
• If you have many factor variables, you might want to use a scatter plot matrix for factors (have a look here)
• You could also go with doing some Parallel coordinates there are several ways to do it in R.
• For a wider range of graphical facilities in R, have a look at the graphics task view.

Now regarding how to do it. One problem with many data points is time till the plot is created. ggplot2, iplots, ggobi are not very good for too many data points (at least from my experience). In which case you might want to focus on R base graphics facilities, or sample your data and on that to use all the other tools. Or you can hope that the people developing iplots extreme (or Acinonyx) would get to an advance release stage.

Good luck,

Tal

-
 Thanks for the links about rflowcyt and Acinonyx. – chl♦ Sep 30 '10 at 8:43 BTW, rflowcyt has been deprecated with recent releases of Bioconductor, it is now recommended to use flowViz. Anyway, both rely on lattice. – chl♦ Sep 30 '10 at 8:53 chl, it is good to know - thank you. – Tal Galili Sep 30 '10 at 13:50 Very thorough answer, Tal! Plot generation time should not be a huge issue. I've been doing most of my graphs with the base package, and the issue of having the graphs look nicer was for when I decide to use a graph for the paper. I had considered using a scatterplot matrix for the numerical variables, but since many of them are of different units (some are in dollars, others in sqft), the only valuable information I'd get would be general trends, but with ~8 numeric variables, an 8x8 SPM is a bit cluttered. – Christopher Aden Sep 30 '10 at 16:49

I'd recommend taking a look at GGobi, which also has an R interface, at least for exploratory purposes. It has a number of graphical displays especially useful for dealing with a large number observations and variables and for linking these together. You might want to start by watching some of the videos under the "Watch a demo" section on the Learn GGobi page.

Update

• DescribeDisplay "R package that provides a way to recreate ggobi graphics in R"
• clusterfly "Explore clustering results in high dimensions"
• rggobi "R package that provides an easy interface with GGobi"
-
@ars Let's add Hadley's R tools to enhance GGobi experience, e.g. DescribeDisplay and clusterfly. – chl Sep 30 '10 at 6:28
Hi ars, as I wrote on my answer - my experience with ggobi is that it doesn't handle large datasets well. Do you have another experience with that ? – Tal Galili Sep 30 '10 at 8:09
@Tal The problem comes from not relying on glyph for screen display/rendering, which is common to R base graphics. This was discussed at the latest DSC conference (j.mp/bpOhBH). Actually, there is an ongoing project with Qt as a backend, and a new port of GGobi, to enhance interactive display of large data sets. – chl Sep 30 '10 at 8:37
@Tal: My experience is that it's quite slow when refreshing/repainting the views, e.g. when adding a variable or dragging to rearrange displays in the PCP. Still, it is usable though not as interactive as advertised with large data. @chl: That's really good to know, thanks! – ars Sep 30 '10 at 17:16
@ars @Tal Here are the links on Qt interface for R (j.mp/d1AJp7) and GGobi (j.mp/cUOvfp). See also Hadley's Github repository! – chl Sep 30 '10 at 17:37
show 1 more comment

Mondrian provides interactive features and handles quite large data sets (it's in Java, though).

Paraview includes 2D/3D viz. features.

-
 Thanks for the two novel libraries. My main conflict with these two is that I am submitting my report via paper copy, so interactive graphics might not be fully used. The graphics of Mondrian look pretty complex. I'll give it a look-see. – Christopher Aden Sep 30 '10 at 15:20 @Christopher For Mondrian, you have the "equivalent" R version through iplots cited by @Tal. About Paraview, you have the option to save a screenshot of your viz. DescribeDisplay is the way to go for exporting dynamic visualization from GGobi, cran.r-project.org/web/packages/DescribeDisplay/index.html. – chl♦ Sep 30 '10 at 15:27

I would like to bring to your attention, Parallel Coordinates: Visual Multidimensional Geometry and Its Applications, which contains the latest breakthroughs and applications in the field.

The book was praised by Stephen Hawking among others. Surfaces are described (using duality) by their normal vectors at its points. It contains applications to Air Traffic Control (Automatic Collision Avoidance -- 3 USA Patents), Multivariate Data Mining (on real datasets some with hundreds of variables), Multiobjective Optimization, Process Control, Intensive Care Smart Displays, Security, Network visualization and recently Big Data.

-
Hi Alfred, thanks for joining the site, this comes off as a bit promotionalist though. Perhaps you have an example similar to the OP's dataset (14 variables and 345,000 observations) that you can provide a picture and describe/demonstrate how parallel coordinates could be useful? Many of the static parallel coordinates charts I have seen with so many observations tend to look like spaghetti on a plate, I suspect you have better insight how to make sense of such big N data though. – Andy W Aug 30 '12 at 12:26
Hi Andy, I do not have an example with that many observations. In – Alfred Inselberg Sep 24 '12 at 21:31
Hi Andy, I do not have an example with that many observations. Interactivity is essential for data exploration. I worked on a dataset with about 800 variables and 10,000 observations in a celllural telephone network to discover "mysterious" failures. Using the classifiers stagewise and with interactive the 11 variables responsible were found and tracing them backwards in time unsual activity in the network was detected 3-4 prior to noticing the failure. – Alfred Inselberg Sep 24 '12 at 21:45