Experimental evidence supporting Tufte-style visualizations?

Question

Q: Does there exist experimental evidence supporting Tufte-style, minimalist, data-speak visualizations over the chart-junked visualizations of, say, Nigel Holmes?

I asked how to add chart-junk to R plots here and responders threw a hefty amount of snark back at me. So, surely, there must be some experimental evidence, to which I'm not privy, supporting their anti-chart junk position---more evidence than just "Tufte said so." Right?

If such evidence exists it would contradict a lot of psychological research we have regarding humans, their memory recall, and pattern identification. So I'd certainly be excited to read about it.

A little anecdote: at a conference I asked Edward Tufte how he regards experimental evidence finding that junk animations and videos improve humans' understanding and memory recall [see research cited in Brain Rules]. His response: "Don't believe them." So much for the scientific method!

P.S. Of course, I'm needling people a little here. I own all of Tufte's books and think his work is incredible. I just think that his supporters have oversold some of his arguments.

NOTE: This is a re-post of a question I asked on StackOverflow. Moderators closed it because it wasn't programming-specific. CrossValidated might be a better home.

UPDATE: There are some useful links in the comments section of my original question post---namely, to the work of Chambers, Cleveland, and the datavis group at Stanford.

UPDATE: This question deals with similar subject matter.

Answer 1

The literature is vast. Experimental evidence is abundant but incomplete. For an introduction that focuses on the psychological and semiotic investigations, see Alan M. MacEachren, How Maps Work (1995; 2004 in paperback). Jump directly to chapter 9 (near the end) and then work backwards through any preliminary material that interests you. The bibliography is extensive (over 400 documents) but is getting a little long in the tooth. Although the title suggests a focus on cartography, most of the book is relevant to how humans create meaning out of and interpret graphical information.

Don't expect to get a definitive answer out of any amount of such research. Remember that Tufte, Cleveland, and others were primarily focused on creating graphics that enable (above all) accurate, insightful communication of and interpretation of data. Other graphics artists and researchers have other aims, such as influencing people, creating effective propaganda, simplifying complex datasets, and expressing their artistic sensibilities within a graphical medium. These are almost diametrically opposed to the first set of objectives, whence the hugely differing approaches and recommendations you will find.

Given this, I think a review of Cleveland's research should be sufficiently convincing that many of Tufte's design recommendations have decent experimental justification. These include his use of the Lie Factor, the Data-Ink Ratio, small multiples, and chartjunk for critically evaluating and designing statistical graphics.

Answer 2

Here's some;

• Cleveland and McGill (1984, JASA) Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods
• Cleveland and McGill (1987, JRSSA) Graphical Perception: The Visual Decoding of Quantitative Information on Graphical Displays of Data
• Lewandowsky and Spence (1989) Discriminating Strata in Scatterplots
• Spence and Lewandowsky (1991) Displaying Proportions and Percentages
• Spence Kutlesa and Rose (1999) Using Color to Code Quantity in Spatial Displays

Ask the Google for the full references

Answer 3

It's worth remembering that information visualisation isn't some island cut off from all other forms of visual communication. If you want to produce work based on evidence based princples, I'd argue it's best to look where the evidence is strongest.

I've read specific research on data visualisation techniques, and general research in cognitive science and in general design research, and I find that thinking through how the more powerful, more thorough general research applies to each brief and each element used is often more effective and useful than trying to apply the narrowly applied field-specific research which often suffers from small samples, weak research techniques, narrow investigation and/or deeply ingrained assumptions.

There are two excellent books I recommend as an introduction, one with the science as a starting point, one with general principles as a starting point, bringing in evidence:

• Vision Science by Steve Palmer. It's a beast, and as a student I nearly gave myself a back injury on the few occasions I was foolish enough to carry it in a backpack, but it's also possibly the best science textbook I've ever seen, and a great example of crisp visual and verbal communciation itself. I went through it recently to label out the chapters with content directly relevant to my work in visualisation and information design, expecting to only label a few: I ended up labelling every chapter except one.
• Universal Principles of Design by Rockport Press. A very ambitious and useful book which crunches cognitive science research with case studies and examples from across all branches of design into a series of awesomely clear and straight to the point double-page spreads, each covering one established, evidence based and practical principle, with practical suggestions, worked examples and suggested further reading. Very stimulating, so long as you think of it as a list of tools with suggested uses not a list of rules.

The only downside is, this approach takes more thinking to see how such principles are applicable. If you're looking for a list of arbitrary rules, as many in the data vis community seem to be, I'd say there isn't one and never will be except where people make massive unjustified assumptions and generalisations, or make things up. The better quality applied research is useful, but it helps to have a solid framework which it can slot into.

Most of Tufte's general principles such as data-ink and chart-junk can be traced back to solid general principles such as signal-noise ratios, figure-ground, attenuation, and others - but on the route to becoming field-specific and prescriptive, they have been combined with hefty assumptions and generalisations about your objectives and audience that turn them into blunt tools. Many of the apparent contradictions and debates in the applied research aren't contradictions at all if you take a step back, take context into account and work through from the underlying core principles and the particular features of each case.

Answer 4

There was one really good study in the field of cartography (Hegarty et al. (2009): Naïve cartography: How intuitions about display configuration can hurt performance. Published in: Cartographica The International Journal for Geographic Information and Geovisualization 44(3):171-186)

It is especially interesting as the authors looked at a more complex task than simply reading values of a bar chart: Expert and novice users had to determine wind speeds and pressure gradients from a meterological map. Both groups of participants intuitively preferred a map with added relief shading and state borders (something Tufte would probably refer to as “chartjunk” as it is irrelevant to the task) against a more minimalistic map showing only the outline of America in the background. But even though there was such a strong personal preference for chartjunk, participants actually performed significantly worse using this map, both in terms of accuracy and response time.

What I found particularly interesting about this study is that a complex use case (like meterology) is really common for us designers/cartographers/data analyst. Often times it is not just about some little bar chart, but we need to design entire dashboards, thematic maps, Sankey diagrams,... Cutting down on your chart junk does have improve your visualizations in this context a lot of times.