Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 13 down vote favorite
9

I'm working on a project that involves 14 variables and 345,000 observations for housing data (things like year built, square footage, price sold, county of residence, etc). I'm concerned with trying to find good graphical techniques and R libraries that contain nice plotting techniques.

I'm already seeing what in ggplot and lattice will work nicely, and I'm thinking of doing violin plots for some of my numerical variables.

What other packages would people recommend for displaying a large amount of either numerical or factor-typed variables in a clear, polished, and most importantly, succinct manner?

share|improve this question
"clear, polished, and most importantly, succinct manner" Sounds like ggplot2 to me. – Brandon Bertelsen Sep 30 '10 at 4:49
1  
I'm looking not just for R libraries, but also any specific types of graphs. My knowledge of graphs is limited to scatter, box, qq, histograms, violins, kernel density estimations, etc. Any slightly more obscure graphs that can reveal more about the data than those would be fantastic. – Christopher Aden Sep 30 '10 at 4:52
2  
parallel coordinates sounds like another one to mention. Also methods of dimensionality reduction might be helpful. – Tal Galili Sep 30 '10 at 8:09

5 Answers

up vote 10 down vote accepted

The best "graph" is so obvious nobody has mentioned it yet: make maps. Housing data depend fundamentally on spatial location (according to the old saw about real estate), so the very first thing to be done is to make a clear detailed map of each variable. To do this well with a third of a million points really requires an industrial-strength GIS, which can make short work of the process. After that it makes sense to go on and make probability plots and boxplots to explore univariate distributions, and to plot scatterplot matrices and wandering schematic boxplots, etc, to explore dependencies--but the maps will immediately suggest what to explore, how to model the data relationships, and how to break up the data geographically into meaningful subsets.

share|improve this answer
Great idea! I have latitudes and longitudes of all datapoints already, so such a task would be relatively elementary. I was thinking the maps library would be a good way to go, unless there's something better. – Christopher Aden Sep 30 '10 at 20:43
2  
@Christopher You can also do this with ggplot2 (esp. if you don't need to draw country boundaries), had.co.nz/ggplot2/coord_map.html. Otherwise, maps, gmaps are better. There's also GeoXp and an R interface to GRASS. BTW, Mondrian has a plugin for geographical data :) – chl♦ Sep 30 '10 at 21:12
Assigning a best answer can be difficult when there's several great suggestions, but I feel this is the right direction, keeping "succinct" in mind. I will give ggplot2 a try, and take a look at maps, GeoXp, and Mondrian. Thanks for the idea of graphing spatially! – Christopher Aden Oct 1 '10 at 17:34
I've had good success with lattice's levelplot and contourplot. Package fields has some nice features, including quiltplot which is nice if your data is not strictly on a grid. It also has a nice thin-plate spline function Tps to smooth non-gridded data to a grid. As for dedicated GIS software, GRASS somehow never makes sense to me, I prefer QGIS. – Wayne Sep 2 '12 at 13:37
up vote 5 down vote

I feel you are actually asking two questions: 1) what types of visualizations to use and 2) what R package can produce them.

In the case of what type of graph to use, there are many, and it depends on your needs (e.g: types of variables - numeric, factor, geographic etc, and the type of connections you are interested to display):

  • If you have many numeric variables, you might want to use a scatter plot matrix (have a look here)
  • If you have many factor variables, you might want to use a scatter plot matrix for factors (have a look here)
  • You could also go with doing some Parallel coordinates there are several ways to do it in R.
  • For a wider range of graphical facilities in R, have a look at the graphics task view.

Now regarding how to do it. One problem with many data points is time till the plot is created. ggplot2, iplots, ggobi are not very good for too many data points (at least from my experience). In which case you might want to focus on R base graphics facilities, or sample your data and on that to use all the other tools. Or you can hope that the people developing iplots extreme (or Acinonyx) would get to an advance release stage.

Good luck,

Tal

share|improve this answer
Thanks for the links about rflowcyt and Acinonyx. – chl♦ Sep 30 '10 at 8:43
BTW, rflowcyt has been deprecated with recent releases of Bioconductor, it is now recommended to use flowViz. Anyway, both rely on lattice. – chl♦ Sep 30 '10 at 8:53
chl, it is good to know - thank you. – Tal Galili Sep 30 '10 at 13:50
Very thorough answer, Tal! Plot generation time should not be a huge issue. I've been doing most of my graphs with the base package, and the issue of having the graphs look nicer was for when I decide to use a graph for the paper. I had considered using a scatterplot matrix for the numerical variables, but since many of them are of different units (some are in dollars, others in sqft), the only valuable information I'd get would be general trends, but with ~8 numeric variables, an 8x8 SPM is a bit cluttered. – Christopher Aden Sep 30 '10 at 16:49
up vote 5 down vote

I'd recommend taking a look at GGobi, which also has an R interface, at least for exploratory purposes. It has a number of graphical displays especially useful for dealing with a large number observations and variables and for linking these together. You might want to start by watching some of the videos under the "Watch a demo" section on the Learn GGobi page.

Update

Links to Hadley Wickham's tools for GGobi, as suggested by chl in the comments:

  • DescribeDisplay "R package that provides a way to recreate ggobi graphics in R"
  • clusterfly "Explore clustering results in high dimensions"
  • rggobi "R package that provides an easy interface with GGobi"
share|improve this answer
1  
@ars Let's add Hadley's R tools to enhance GGobi experience, e.g. DescribeDisplay and clusterfly. – chl♦ Sep 30 '10 at 6:28
Hi ars, as I wrote on my answer - my experience with ggobi is that it doesn't handle large datasets well. Do you have another experience with that ? – Tal Galili Sep 30 '10 at 8:09
@Tal The problem comes from not relying on glyph for screen display/rendering, which is common to R base graphics. This was discussed at the latest DSC conference (j.mp/bpOhBH). Actually, there is an ongoing project with Qt as a backend, and a new port of GGobi, to enhance interactive display of large data sets. – chl♦ Sep 30 '10 at 8:37
1  
@Tal: My experience is that it's quite slow when refreshing/repainting the views, e.g. when adding a variable or dragging to rearrange displays in the PCP. Still, it is usable though not as interactive as advertised with large data. @chl: That's really good to know, thanks! – ars Sep 30 '10 at 17:16
1  
@ars @Tal Here are the links on Qt interface for R (j.mp/d1AJp7) and GGobi (j.mp/cUOvfp). See also Hadley's Github repository! – chl♦ Sep 30 '10 at 17:37
show 1 more comment
up vote 2 down vote

Mondrian provides interactive features and handles quite large data sets (it's in Java, though).

Paraview includes 2D/3D viz. features.

share|improve this answer
Thanks for the two novel libraries. My main conflict with these two is that I am submitting my report via paper copy, so interactive graphics might not be fully used. The graphics of Mondrian look pretty complex. I'll give it a look-see. – Christopher Aden Sep 30 '10 at 15:20
@Christopher For Mondrian, you have the "equivalent" R version through iplots cited by @Tal. About Paraview, you have the option to save a screenshot of your viz. DescribeDisplay is the way to go for exporting dynamic visualization from GGobi, cran.r-project.org/web/packages/DescribeDisplay/index.html. – chl♦ Sep 30 '10 at 15:27
up vote -3 down vote

I would like to bring to your attention, Parallel Coordinates: Visual Multidimensional Geometry and Its Applications, which contains the latest breakthroughs and applications in the field.

The book was praised by Stephen Hawking among others. Surfaces are described (using duality) by their normal vectors at its points. It contains applications to Air Traffic Control (Automatic Collision Avoidance -- 3 USA Patents), Multivariate Data Mining (on real datasets some with hundreds of variables), Multiobjective Optimization, Process Control, Intensive Care Smart Displays, Security, Network visualization and recently Big Data.

share|improve this answer
5  
Hi Alfred, thanks for joining the site, this comes off as a bit promotionalist though. Perhaps you have an example similar to the OP's dataset (14 variables and 345,000 observations) that you can provide a picture and describe/demonstrate how parallel coordinates could be useful? Many of the static parallel coordinates charts I have seen with so many observations tend to look like spaghetti on a plate, I suspect you have better insight how to make sense of such big N data though. – Andy W Aug 30 '12 at 12:26
Hi Andy, I do not have an example with that many observations. In – Alfred Inselberg Sep 24 '12 at 21:31
Hi Andy, I do not have an example with that many observations. Interactivity is essential for data exploration. I worked on a dataset with about 800 variables and 10,000 observations in a celllural telephone network to discover "mysterious" failures. Using the classifiers stagewise and with interactive the 11 variables responsible were found and tracing them backwards in time unsual activity in the network was detected 3-4 prior to noticing the failure. – Alfred Inselberg Sep 24 '12 at 21:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.