25
$\begingroup$

I'd like to show how the values of certain variables (~ 15) change over time, but I'd also like to show how the variables differ from each other in each year. So I created this plot:

enter image description here

But even when changing the colour scheme or adding different line/shape types this looks messy. Is there a better way to visualise this kind of data?

Test data with R code:

structure(list(Var = structure(c(1L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 6L, 
6L, 6L, 6L, 6L, 6L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 8L, 8L, 8L, 8L, 
8L, 8L, 8L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 11L, 11L, 11L, 11L, 11L, 
11L, 11L, 12L, 12L, 12L, 12L, 12L, 12L, 13L, 14L, 14L, 14L, 14L, 
14L, 14L, 14L, 16L, 16L, 16L, 16L, 16L, 16L, 17L, 17L, 17L, 17L, 
17L, 17L, 17L, 18L, 18L, 18L, 18L, 18L, 18L, 18L), .Label = c("A", 
"B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M", "N", 
"O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", "Y", "Z"), class = "factor"), 
    Year = c(2015L, 1991L, 1993L, 1996L, 2000L, 2004L, 2011L, 
    2015L, 1991L, 1993L, 1996L, 2000L, 2004L, 2011L, 2015L, 1991L, 
    1993L, 1996L, 2000L, 2004L, 2011L, 2015L, 1993L, 1996L, 2000L, 
    2004L, 2011L, 2015L, 1991L, 1993L, 1996L, 2000L, 2004L, 2011L, 
    2015L, 1991L, 1993L, 1996L, 2000L, 2004L, 2011L, 2015L, 1991L, 
    1993L, 1996L, 2000L, 2004L, 2011L, 2015L, 1991L, 1993L, 1996L, 
    2000L, 2004L, 2011L, 2015L, 1993L, 1996L, 2000L, 2004L, 2011L, 
    2015L, 2015L, 1991L, 1993L, 1996L, 2000L, 2004L, 2011L, 2015L, 
    1991L, 1993L, 1996L, 2000L, 2011L, 2015L, 1991L, 1993L, 1996L, 
    2000L, 2004L, 2011L, 2015L, 1991L, 1993L, 1996L, 2000L, 2004L, 
    2011L, 2015L), Val = c(25.6, 22.93, 20.82, 24.1, 24.5, 29, 
    25.55, 24.5, 24.52, 20.73, 25.8, 25.5, 29.5, 27.7, 25.1, 
    25, 24.55, 26.75, 25, 30.5, 27.25, 25.1, 22.4, 27.07, 26, 
    29, 27.2, 24.2, 23, 24.27, 27.68, 27, 30.5, 28.1, 24.9, 23.75, 
    22.75, 27.25, 25, 29, 28.45, 24, 20.25, 17.07, 24.45, 25, 
    28.5, 26.75, 24.9, 21.25, 20.65, 25.1, 24.5, 26.5, 25.35, 
    23.5, 21.93, 26.5, 24.5, 29, 29.1, 26.4, 28.1, 23.75, 26.5, 
    28.05, 27, 30.5, 25.65, 23.3, 23.25, 24.57, 26.07, 27.5, 
    28.85, 27.7, 22, 23.43, 26.88, 27, 30.5, 29.25, 28.1, 23, 
    23.8, 28.32, 27, 29.5, 29.15, 27.6)), row.names = c(1L, 4L, 
5L, 6L, 7L, 8L, 9L, 10L, 13L, 14L, 15L, 16L, 17L, 18L, 19L, 20L, 
21L, 22L, 23L, 24L, 25L, 26L, 27L, 28L, 29L, 30L, 31L, 32L, 35L, 
36L, 37L, 38L, 39L, 40L, 41L, 44L, 45L, 46L, 47L, 48L, 49L, 50L, 
53L, 54L, 55L, 56L, 57L, 58L, 59L, 62L, 63L, 64L, 65L, 66L, 67L, 
68L, 69L, 70L, 71L, 72L, 73L, 74L, 75L, 78L, 79L, 80L, 81L, 82L, 
83L, 84L, 87L, 88L, 89L, 90L, 91L, 92L, 95L, 96L, 97L, 98L, 99L, 
100L, 101L, 104L, 105L, 106L, 107L, 108L, 109L, 110L), na.action = structure(c(2L, 
3L, 11L, 12L, 33L, 34L, 42L, 43L, 51L, 52L, 60L, 61L, 76L, 77L, 
85L, 86L, 93L, 94L, 102L, 103L), .Names = c("2", "3", "11", "12", 
"33", "34", "42", "43", "51", "52", "60", "61", "76", "77", "85", 
"86", "93", "94", "102", "103"), class = "omit"), class = "data.frame", .Names = c("Var", 
"Year", "Val"))
$\endgroup$
  • 2
    $\begingroup$ Can you post the data? It's easy enough to find roughly similar examples, but to keep the thread tied together, people having the same sandbox to play in would help. Also, what's the significance of the green zone? $\endgroup$ – Nick Cox Jan 11 '16 at 13:33
  • $\begingroup$ See also suggestions in stats.stackexchange.com/questions/126480/… $\endgroup$ – Nick Cox Jan 12 '16 at 12:27
  • $\begingroup$ @NickCox Sure, should have thought of that earlier! I left out the green zone as it's not essential (it only shows the range of values which is considered "sufficient") $\endgroup$ – user45065 Jan 12 '16 at 12:35
41
+100
$\begingroup$

Fortuitously or otherwise, your example is of optimal size (up to 7 values for each of 15 groups) first, to show that there is a problem graphically; and second, to allow other and fairly simple solutions. The graph is of a kind often called spaghetti by people in different fields, although it's not always clear whether that term is meant as affectionate or abusive. The graph does show the collective or family behaviour of all the groups, but it is fairly hopeless at showing the detail to be explored.

One standard alternative is just to show the separate groups in separate panels, but that in turn can make precise group-to-group comparisons difficult; each group is separated from its context of the other groups.

So why not combine both ideas: a separate panel for each group, but also show the other groups as backdrop? This hinges crucially on highlighting the group which is in focus and on downplaying the others, which is easy enough in this example given some use of line colour, thickness etc. In other examples, marker or point symbol choices might be natural instead.

enter image description here

In this case, details of possible practical or scientific importance or interest are highlighted:

  1. We only have one value for A and M.

  2. We don't have all values for all given years in all other cases.

  3. Some groups plot high, some low, and so forth.

I won't attempt an interpretation here: the data are anonymous, but that is the researcher's concern in any case.

Depending on what is easy or possible in your software, there is scope for changing small details here, such as whether axis labels and titles are repeated (there are simple arguments both for and against).

The larger issue is how far this strategy will work more generally. The number of groups is the major driver, more so than the number of points in each group. Roughly speaking, the approach might work up to about 25 groups (a 5 x 5 display, say): with more groups, not only do the graphs become smaller and more difficult to read, but even the researcher loses the inclination to scan all the panels. If there were hundreds (thousands, ...) of groups, it would usually be essential to select a small number of groups to show. Some mix of criteria such as selecting some "typical" and some "extreme" panels would be needed; that should be driven by project goals and some idea of what makes sense for each dataset. Another approach that can be efficient is to emphasize a small number of series in each panel. So, if there were 25 broad groups, each broad group could be shown with all others as backdrop. Alternatively, there could be some averaging or other summarization. Using (e.g.) principal or independent components might also be a good idea.

Although the example calls for line plots, the principle is naturally much general. Examples could be multiplied, scatter plots, model diagnostic plots, etc.

Some references for this approach [others are most welcome]:

Cox, N.J. 2010. Graphing subsets. Stata Journal 10: 670-681.

Knaflic, C.N. 2015. Storytelling with Data: A Data Visualization Guide for Business Professionals. Hoboken, NJ: Wiley.

Koenker, R. 2005. Quantile Regression. Cambridge: Cambridge University Press. See pp.12-13.

Schwabish, J.A. 2014. An economist's guide to visualizing data. Journal of Economic Perspectives 28: 209-234.

Unwin, A. 2015. Graphical Data Analysis with R. Boca Raton, FL: CRC Press.

Wallgren, A., B. Wallgren, R. Persson, U. Jorner, and J.-A. Haaland. 1996. Graphing Statistics and Data: Creating Better Charts. Newbury Park, CA: Sage.

Note: The graph was created in Stata. subsetplot must be installed first with ssc inst subsetplot. Data were copied and pasted from R and value labels were defined to show years as 90 95 00 05 10 15. The main command is

subsetplot connected Val Year, by(Var) c(L) lcolor(gs12) backdrop(line) xtitle("") combine(imargin(small)) subset(lcolor(blue) mcolor(blue))

EDIT Extra references May, September, December 2016; April, June 2017, December 2018, April 2019:

Cairo, A. 2016. The Truthful Art: Data, Charts, and Maps for Communication. San Francisco, CA: New Riders. p.211

Camões, J. 2016. Data at Work: Best Practices for Creating Effective Charts and Information Graphics in Microsoft Excel. San Francisco, CA: New Riders. p.354

Carr, D.B. and Pickle, L.W. 2010. Visualizing Data Patterns with Micromaps. Boca Raton, FL: CRC Press. p.85.

Grant, R. 2019. Data Visualization: Charts, Maps, and Interactive Graphics. Boca Raton, FL: CRC Press. p.52.

Koponen, J. and Hildén, J. 2019. The Data Visualization Handbook. Espoo: Aalto ARTS Books. See p.101.

Kriebel, A. and Murray, E. 2018. #MakeoverMonday: Improving How We Visualize and Analyze Data, One Chart at a Time. Hoboken, NJ: John Wiley. p.303.

Rougier, N.P., Droettboom, M. and Bourne, P.E. 2014. Ten simple rules for better figures. PLOS Computational Biology 10(9): e1003833. doi:10.1371/journal.pcbi.1003833 link here

Schwabish, J. 2017. Better Presentations: A Guide for Scholars, Researchers, and Wonks. New York: Columbia University Press. See p.98.

Wickham, H. 2016. ggplot2: Elegant Graphics for Data Analysis. Cham: Springer. See p.157.

$\endgroup$
  • $\begingroup$ +1, wonderful, is there an R or SAS function that is able to do this type of chart? It's really great. $\endgroup$ – forecaster Jan 12 '16 at 13:34
  • $\begingroup$ I really like this idea! Just wondering about the best way to plot this in R using ggplot2. I'll just wait a bit before I accept the answer, hope that's fine. $\endgroup$ – user45065 Jan 12 '16 at 13:37
  • 2
    $\begingroup$ Sorry, I have no idea about how to do anything in SAS. Surely anything Stata can do, R can do as well or better, or so its users keep telling me.... $\endgroup$ – Nick Cox Jan 12 '16 at 13:40
  • $\begingroup$ @NickCox Not a problem at all, I figured it out, it looks really good and is perfect for my purpose. $\endgroup$ – user45065 Jan 12 '16 at 13:47
  • $\begingroup$ @NickCox, two more references are 1. The Elements of Graphing Data by W.S. Cleveland.A new book, 2. Storytelling with Data: A Data Visualization Guide for Business Professionals by Cole Nussbaumer Knaflic.This book (#2) has a case study chapter called "Strategies for avoiding the spaghetti graph". $\endgroup$ – forecaster Jan 20 '16 at 14:45
21
$\begingroup$

As a complement to Nick's answer, here's some R code for making a similar plot using simulated data:

library(ggplot2)

get_df <- function(label="group A", n_obs=10, drift=runif(1)) {
    df <- data.frame(time=seq(1, n_obs), label=label)
    df$y <- df$time * drift + cumsum(rnorm(n_obs))
    return(df)
}
df_list <- lapply(sprintf("group %s", toupper(letters[1:9])),
                  function(label) { get_df(label) })
df <- do.call(rbind, df_list)
df$label2 <- df$label

p <- (ggplot(df, aes(x=time, y=y, group=label2)) +
      geom_line(size=0.9, alpha=0.8,
                data=df[, c("time", "y", "label2")], color="grey") +
      geom_line(size=1.1, color="black") +
      ylab("") +
      theme_bw() +
      theme(panel.border=element_blank()) +
      theme(strip.background=element_blank()) +
      facet_wrap(~ label))
p
ggsave("example_facet.png", p, width=10, height=8)

example plot

$\endgroup$
6
$\begingroup$

For those wanting to use a ggplot2 approach in R consider the facetshade function in the package extracat. This offers a general approach, not just for line plots. Here is an example with scatterplots (from the foot of this page):

data(olives, package="extracat")
library(scales)
fs1 <- facetshade(data = olives,
                  aes(x = palmitic, y = palmitoleic), f = .~Area)
fs1 + geom_point(colour = alpha("black", 0.05)) +
      geom_point(data = olives, colour = "red") +
      facet_wrap(f=~Area, nrow=3) + theme(legend.position="none")

enter image description here


EDIT: Using Adrian's simulated dataset from his earlier answer:

library(extracat)
facetshade(df, aes(x=time, y=y), f = .~label, bg.all = FALSE, keep.orig = TRUE) +
           geom_line(aes(x=time, y=y, group=orig.label),colour = alpha(1,0.3)) +
           geom_line(data=df, aes(colour=label), size = 1.2) + xlab("") + ylab("")

Another approach is to draw two separate layers, one for the background and one for the highlighted cases. The trick is to draw the background layer using the dataset without the faceting variable. For the olive oil dataset the code is:

data(olives, package="extracat")
ggplot(olives, aes(palmitic, palmitoleic)) + 
  facet_wrap(~Area, nrow=3) + 
  geom_point(data=olives %>% select(-Area), colour=alpha("black", 0.05)) + 
  geom_point(data=olives, colour="red") + 
  theme(legend.position="none")
$\endgroup$
  • 1
    $\begingroup$ This seems like a nice general approach (+1), but the particular example is more relating to a different problem. A bunch of repeated scatter plots with differently highlighted regions is not gonna work for the question which is about time series. $\endgroup$ – Martijn Weterings Dec 18 '18 at 16:36
  • $\begingroup$ @martin Actually it is and that is also Adrian's solution. Note that he uses two identical labelling variables so that one can be dropped in the background layer. The coding idea is more obvious with the tidyverse notation below and, as so often, elegant formatting of graphics can mask the important parts of code. ggplot(df %>% select(-label), aes(x=time, y=y, group=label2)) + geom_line(alpha=0.8, color="grey") + labs(y=NULL) + geom_line(data=df, color="red") + facet_wrap(~ label) $\endgroup$ – Antony Unwin Dec 22 '18 at 14:23
4
$\begingroup$

Here is a solution inspired by Ch. 11.3, the section on "Texas Housing Data", in Hadley Wickham's Book on ggplot2. Here I fit a linear model to each time series , take the residuals (which are centered around mean 0), and draw a summary line in a different color.

library(ggplot2)
library(dplyr)
#works with dplyr version 0.4.3.9000 from Github (hadley/dplyr@4f2d7f8), or higher

df1 <- as.data.frame(list(Var = structure(c(1L, 2L, 2L, 2L, 2L, 2L, 2L, 
                                 2L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 6L, 
                                 6L, 6L, 6L, 6L, 6L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 8L, 8L, 8L, 8L, 
                                 8L, 8L, 8L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 11L, 11L, 11L, 11L, 11L, 
                                 11L, 11L, 12L, 12L, 12L, 12L, 12L, 12L, 13L, 14L, 14L, 14L, 14L, 
                                 14L, 14L, 14L, 16L, 16L, 16L, 16L, 16L, 16L, 17L, 17L, 17L, 17L, 
                                 17L, 17L, 17L, 18L, 18L, 18L, 18L, 18L, 18L, 18L), .Label = c("A", 
                                                                                               "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M", "N", 
                                                                                               "O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", "Y", "Z"), class = "factor"), 
               Year = c(2015L, 1991L, 1993L, 1996L, 2000L, 2004L, 2011L, 
                        2015L, 1991L, 1993L, 1996L, 2000L, 2004L, 2011L, 2015L, 1991L, 
                        1993L, 1996L, 2000L, 2004L, 2011L, 2015L, 1993L, 1996L, 2000L, 
                        2004L, 2011L, 2015L, 1991L, 1993L, 1996L, 2000L, 2004L, 2011L, 
                        2015L, 1991L, 1993L, 1996L, 2000L, 2004L, 2011L, 2015L, 1991L, 
                        1993L, 1996L, 2000L, 2004L, 2011L, 2015L, 1991L, 1993L, 1996L, 
                        2000L, 2004L, 2011L, 2015L, 1993L, 1996L, 2000L, 2004L, 2011L, 
                        2015L, 2015L, 1991L, 1993L, 1996L, 2000L, 2004L, 2011L, 2015L, 
                        1991L, 1993L, 1996L, 2000L, 2011L, 2015L, 1991L, 1993L, 1996L, 
                        2000L, 2004L, 2011L, 2015L, 1991L, 1993L, 1996L, 2000L, 2004L, 
                        2011L, 2015L), 
               Val = c(25.6, 22.93, 20.82, 24.1, 24.5, 29, 
                       25.55, 24.5, 24.52, 20.73, 25.8, 25.5, 29.5, 27.7, 25.1, 
                       25, 24.55, 26.75, 25, 30.5, 27.25, 25.1, 22.4, 27.07, 26, 
                       29, 27.2, 24.2, 23, 24.27, 27.68, 27, 30.5, 28.1, 24.9, 23.75, 
                       22.75, 27.25, 25, 29, 28.45, 24, 20.25, 17.07, 24.45, 25, 
                       28.5, 26.75, 24.9, 21.25, 20.65, 25.1, 24.5, 26.5, 25.35, 
                       23.5, 21.93, 26.5, 24.5, 29, 29.1, 26.4, 28.1, 23.75, 26.5, 
                       28.05, 27, 30.5, 25.65, 23.3, 23.25, 24.57, 26.07, 27.5, 
                       28.85, 27.7, 22, 23.43, 26.88, 27, 30.5, 29.25, 28.1, 23, 
                       23.8, 28.32, 27, 29.5, 29.15, 27.6)), 
               row.names = c(1L, 4L, 
                           5L, 6L, 7L, 8L, 9L, 10L, 13L, 14L, 15L, 16L, 17L, 18L, 19L, 20L, 
                           21L, 22L, 23L, 24L, 25L, 26L, 27L, 28L, 29L, 30L, 31L, 32L, 35L, 
                           36L, 37L, 38L, 39L, 40L, 41L, 44L, 45L, 46L, 47L, 48L, 49L, 50L, 
                           53L, 54L, 55L, 56L, 57L, 58L, 59L, 62L, 63L, 64L, 65L, 66L, 67L, 
                           68L, 69L, 70L, 71L, 72L, 73L, 74L, 75L, 78L, 79L, 80L, 81L, 82L, 
                           83L, 84L, 87L, 88L, 89L, 90L, 91L, 92L, 95L, 96L, 97L, 98L, 99L, 
                           100L, 101L, 104L, 105L, 106L, 107L, 108L, 109L, 110L), 
               na.action = structure(c(2L, 
                          3L, 11L, 12L, 33L, 34L, 42L, 43L, 51L, 52L, 60L, 61L, 76L, 77L, 
                          85L, 86L, 93L, 94L, 102L, 103L), 
                .Names = c("2", "3", "11", "12","33", "34", "42", "43", "51", "52", "60", 
                           "61", "76", "77", "85", "86", "93", "94", "102", "103"), class = "omit"), 
                class = "data.frame", .Names = c("Var","Year", "Val"))


df1 %>%
        group_by(Var) %>%
        do(mutate(.,resid = resid(lm(Val ~ Year, data=., na.action = na.exclude)))) %>%
        ggplot(aes(Year, resid)) +
        labs(y=paste0("Val "), x="Year") +
        geom_line(aes(group = Var), alpha = 1/5) +
        geom_line(stat = "summary", fun.y = "mean", colour = "red")

enter image description here

$\endgroup$
  • 1
    $\begingroup$ The main idea here seems to be that you could add a summary curve of some kind to help the eye and mind. Agreed, but in your answer you could spell out the trade-off on shifting to mean (or reference level) 0 rather than leaving the original units and values. Subject-matter experts and/or clients might well think in terms of 24 or 28 or whatever values are. Naturally the data here are just a vehicle for the discussion but the point is highly general. $\endgroup$ – Nick Cox Jan 15 '16 at 9:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy