2
$\begingroup$

Simple scatter plots or a correlation matrix can be misleading since they aggregate all years together. What would be te best way to visualise relationships between variables taking time into account? For example, the scatter matrix below shows relations between certain variables, but aggregates the data for all the years. enter image description here

What are common ways to visualise relations for panel data? Is it simply to make multiple scatter plots for each year?

The data set is a panel data set containing data for 10 countries for 20 years. I created some sample data with:

# Sample data frame
df_Q <- data.frame(country = rep(c("A","B","C","D","E","F","G","H","I","J"),each=20),
               year = rep(2001:2020, times = 10),
               gini = rnorm(200, mean = 25, sd = 7),
               total_trade = rnorm(200, mean = 14, sd = 8), 
               intraEU_trade = rnorm(200, mean = 7, sd = 6),
               inward_FDI = rnorm(200, mean = 10, sd = 5),
               outward_FDI = rnorm(200, mean = 20, sd = 4),
               unemployment = rnorm(200, mean = 15, sd = 10))

The data looks like this:

  # country year     gini total_trade intraEU_trade inward_FDI outward_FDI unemployment
# 1       A 2001 21.94901   22.717850      9.536230  14.539323    22.84906    20.408894
# 2       A 2002 12.77493   15.837161      8.702562  12.178246    16.76624     9.992739
# 3       A 2003 30.79006    0.750060      3.924690   7.463981    24.93762    25.092569
# 4       A 2004 24.92870   15.447970     12.882917  13.696632    21.89461    20.729777
# 5       A 2005 31.19249    8.476526     10.160960  10.759731    20.35886    25.562575
# 6       A 2006 20.23424   18.060654      8.271882  13.904214    22.95711    16.132112

I estimated some models with the plm package, but I would like to know how to visualise the bivariate relationships between the variables, taking the year into account.

The plot was created with:

library(corrmorant) 
library(ggplot2)
ggcorrm(df_Q[,c("gini", "total_trade", "intraEU_trade", "inward_FDI", "outward_FDI", "unemployment")]) +
  lotri(geom_point(alpha = 0.7, size = 0.2)) + 
  lotri(geom_smooth(method = "lm", size = 0.6, color = "#ce5348")) + 
  utri_corrtext(nrow = 2, corr_size=FALSE, size=4) +
  dia_names(y_pos = 0.15, size = 2.5) +
  dia_density(lower = 0.3, color = 1, alpha=0.7, fill = "#a2a2a2") + 
  scale_y_continuous(n.breaks = 3) +
  scale_x_continuous(n.breaks = 3) +
  theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1))
$\endgroup$
3
  • 2
    $\begingroup$ Is it possible to access these data for people to work with? Is this a publicly available dataset? Can you provide or make one? How many years do you have? Is it 1 datum / year? What are the statistical units here, countries / firms / something else? $\endgroup$ May 12, 2022 at 20:12
  • 1
    $\begingroup$ I added some sample data and the code I used to create the plot. $\endgroup$ May 12, 2022 at 20:51
  • $\begingroup$ You may need to create a for loop for each year and the combine them. Depending of which dimension is greater (time or group) or what's your main goal, you can add them accordingly. $\endgroup$ May 13, 2022 at 15:47

2 Answers 2

5
$\begingroup$

Specific to panel data: You may want to assign color to year. Since year has a natural ordering, choose a sequential color scheme so that it's easy to tell which points are from earlier vs later years. If you wanted to group by something like country instead, it would be fine to use a qualitative color scheme, in which there's no natural ordering to the colors and they are just chosen to be easy to tell apart.

Not specific to panel data: I like the 2 main approaches laid out in William Cleveland's The Elements of Graphing Data. In general, when trying to visualize how a relationship between 2 quantitative variables depends on a 3rd variable:

  • "superpose" the groups: show a scatterplot, but use another visual variable like color, point shape, etc. to show the different groups. Regarding your example above, I haven't worked with ggcorrm before but presumably you could add mapping=aes(color=year) somewhere.
  • "juxtapose" the groups: show the same plot repeated several times, once for each value of the grouping variable. In ggplot terms, just facet on the year (assuming ggcorrm allows it).

On p.59 of Rafe Donahue's "Fundamental Statistical Concepts in Presenting Data" there's a nice extension of the "juxtapose" idea: Make each facet also show data from the other facets, just greyed out. This essentially highlights one group at a time, but keeps it in context of all the other groups.
Update---here's an example of how to do this in ggplot:

# Generate fake data, with a trend in `total_trade` over time
df_Q <- data.frame(country = rep(c("A","B","C","D","E","F","G","H","I","J"),each=20),
                   year = rep(2001:2020, times = 10),
                   gini = rnorm(200, mean = 25, sd = 7))
df_Q$total_trade = rnorm(200, mean = 14, sd = 8) + 2*(df_Q$year-2000)

# Make a version of the dataset without `year`
df_NoYear <- df_Q
df_NoYear$year <- NULL

# Add `geom_point()` twice:
# FIRST as grey points in the dataset without `year`
# (so it repeats as background in every panel)
# THEN the real data with `year`
library(ggplot2)
ggplot(df_Q, aes(x = gini, y = total_trade)) +
  geom_point(data = df_NoYear, color = "grey85") +
  geom_point() +
  facet_wrap(~ year)

FacetsWithGreyBackground

I hope this illustrates how the "all-other-years" background points can be helpful as reference points, making it easier to notice a subtle trend across facets.

$\endgroup$
8
  • $\begingroup$ Thank you! What I found useful as well is to let the transparency level (alpha in ggplot) depend on the year (more opaque dots represent more recent years for example). $\endgroup$ May 18, 2022 at 9:41
  • 1
    $\begingroup$ No problem! And yes, transparency definitely works too. I've also updated the post with an example of the "all-other-years" background points in each facet. Not sure if this can work with ggcorm specifically, but it's a useful trick for panel data in general. $\endgroup$
    – civilstat
    May 18, 2022 at 12:55
  • 1
    $\begingroup$ More references on backdrop of other facets in grey at stats.stackexchange.com/questions/190152/… The name "front-and-back plot" is used at journals.sagepub.com/doi/full/10.1177/1536867X211025838 $\endgroup$
    – Nick Cox
    May 20, 2022 at 0:33
  • $\begingroup$ @NickCox Oh cool, thanks for the references! I like "front-and-back plot." I think Rafe Donahue's notes called it a "you-are-here plot." So many names for an idea that's useful but too-rarely used. $\endgroup$
    – civilstat
    May 20, 2022 at 0:51
  • $\begingroup$ I read Rafe's notes some years ago. but they didn't stick as a reference to use later. The idea may be much older than I know, but use of grey or some other light colour has not been easy forever. $\endgroup$
    – Nick Cox
    May 20, 2022 at 1:02
3
$\begingroup$

I am hesitant to claim this is the way, or the best way, to visualize panel data like yours, but I can throw out a few suggestions.

You are looking at a (combined) correlation matrix and scatterplot matrix. Your primary concern is that any two variables are confounded with time. Of course, you can look at subsets of the data that hold time constant, but you'd need 20 pairs of matrices (or more in other cases) and that's just not workable.

A rather simple, and presumably workable, option would be to borrow the notion behind coplots (see also: coplot in R or this pdf) and examine these matrices in partially overlapping temporal strata. Since you have 20 years, three strata (1-10, 6-15, 11-20) seems doable.

library(car)  # we'll need this package for enhanced scatterplot matrices
set.seed(1)   # you need to set the seed to generate reproducible data
d = data.frame(country = rep(c("A","B","C","D","E","F","G","H","I","J"),each=20),
               ... )

round(cor(d[,3:8]), 2)
#                gini total_trade intraEU_trade inward_FDI outward_FDI unemployment
# gini           1.00       -0.02          0.07       0.07        0.02        -0.07
# total_trade   -0.02        1.00         -0.03      -0.01        0.06        -0.02
# intraEU_trade  0.07       -0.03          1.00      -0.04        0.02        -0.01
# inward_FDI     0.07       -0.01         -0.04       1.00        0.04        -0.01
# outward_FDI    0.02        0.06          0.02       0.04        1.00        -0.07
# unemployment  -0.07       -0.02         -0.01      -0.01       -0.07         1.00
windows()
  scatterplotMatrix(d[,3:8])

enter image description here

d.early = d[d$year<2011,]
d.mid   = d[d$year>2005 & d$year<2016,]
d.late  = d[d$year>2010,]
round(cor(d.early[,3:8]), 2)
#                gini total_trade intraEU_trade inward_FDI outward_FDI unemployment
# gini           1.00       -0.07          0.14       0.08       -0.02        -0.15
# total_trade   -0.07        1.00         -0.04       0.03        0.05        -0.04
# intraEU_trade  0.14       -0.04          1.00      -0.04       -0.04        -0.06
# inward_FDI     0.08        0.03         -0.04       1.00       -0.06        -0.07
# outward_FDI   -0.02        0.05         -0.04      -0.06        1.00        -0.06
# unemployment  -0.15       -0.04         -0.06      -0.07       -0.06         1.00
windows()
  scatterplotMatrix(d.early[,3:8], main="2001 - 2010")

enter image description here

round(cor(d.mid[,3:8]), 2)
#                gini total_trade intraEU_trade inward_FDI outward_FDI unemployment
# gini           1.00       -0.02          0.10       0.17        0.05        -0.02
# total_trade   -0.02        1.00          0.07       0.05       -0.05        -0.12
# intraEU_trade  0.10        0.07          1.00      -0.01        0.10        -0.13
# inward_FDI     0.17        0.05         -0.01       1.00       -0.07         0.03
# outward_FDI    0.05       -0.05          0.10      -0.07        1.00        -0.12
# unemployment  -0.02       -0.12         -0.13       0.03       -0.12         1.00
windows()
  scatterplotMatrix(d.mid[,3:8], main="2006 - 2015")

enter image description here

round(cor(d.late[,3:8]), 2)
#                gini total_trade intraEU_trade inward_FDI outward_FDI unemployment
# gini           1.00        0.04         -0.01       0.05        0.07         0.01
# total_trade    0.04        1.00         -0.01      -0.05        0.07         0.02
# intraEU_trade -0.01       -0.01          1.00      -0.03        0.07         0.04
# inward_FDI     0.05       -0.05         -0.03       1.00        0.12         0.05
# outward_FDI    0.07        0.07          0.07       0.12        1.00        -0.08
# unemployment   0.01        0.02          0.04       0.05       -0.08         1.00
windows()
  scatterplotMatrix(d.late[,3:8], main="2011 - 2020")

enter image description here


Given that your concern is that both variables are changing over time, a different approach is just to control for time as a third variable. The correlation between two variables after having partialled out a third variable is the partial correlation. It is the correlation between the residuals of regressing each of the variables on the third. A scatterplot of the two variables after having partialled out the third is an added variable plot. It is the scatterplot of the two sets of residuals just mentioned. You can get matrices of both, just as usual. This is quite simple in your case.

d.pcor = lapply(d[,3:8], function(j){  resid(lm(j~d$year))  })
d.pcor = do.call("cbind", d.pcor)
d.pcor = as.data.frame(d.pcor)
names(d.pcor) = names(d)[3:8]

round(cor(d.pcor), 2)
#                gini total_trade intraEU_trade inward_FDI outward_FDI unemployment
# gini           1.00       -0.02          0.07       0.07        0.02        -0.07
# total_trade   -0.02        1.00         -0.03      -0.01        0.06        -0.02
# intraEU_trade  0.07       -0.03          1.00      -0.04        0.01        -0.01
# inward_FDI     0.07       -0.01         -0.04       1.00        0.04        -0.01
# outward_FDI    0.02        0.06          0.01       0.04        1.00        -0.07
# unemployment  -0.07       -0.02         -0.01      -0.01       -0.07         1.00
windows()
  scatterplotMatrix(d.pcor, main="added variable plots")

enter image description here


I want to note, however, that I think your situation is more complicated than just taking out the effect of time. Your data are confounded with time, but they are also non-independent in that they are nested within statistical units (i.e., countries here) that are repeatedly measured.

When dealing with longitudinal / panel data, it is common to make a 'spaghetti' plot. This is usually done for the response variable as a function of time. Note that the name is pejorative. I'm mixed about these, but I usually do make one, possibly with some augmentations, and in conjunction with other plots. (See also @NickCox's excellent answer to Visualising many variables in one plot, and Plotting and presenting longitudinal data, options?)

d.w = reshape(d, direction="wide", idvar="country", timevar="year", 
              v.names=names(d)[3:8]) 
rownames(d.w) = NULL
d.w = d.w[,c("country", sort(names(d.w)[2:121]))]

windows()
  plot(0,0, col="white", xlim=c(1,20), ylim=range(d$total_trade), xlab="year", 
       ylab="total_trade", axes=FALSE, main="Spaghetti plot");  box()
  axis(side=1, at=1:20, labels=paste0("'", sprintf("%02d", 1:20)), cex.axis=.7)
  axis(side=2, at=seq(-10, 35, by=5))
  for(i in 1:10){
    lines(1:20, d.w[i,82:101], lwd=2, col=rgb(0,0,0,alpha=.5))
  }

enter image description here

You have time-varying covariates, so you could make one of these for every variable, I suppose. However, you seem interested in the correlations / scatterplots of each of the different variables relative to each other. In which case, you could extend this idea to have 2-dimensional spaghetti plots:

spaghetti.in.2d = function(x,y){  
  plot(0,0, col="white", xlim=range(d[,x]), ylim=range(d[,y]),
       xlab=x, ylab=y, main="2 dimensions of spaghetti")
  for(i in 1:10){
    xs = unlist(d.w[i,sort(grep(x, names(d.w)))])
    ys = unlist(d.w[i,sort(grep(y, names(d.w)))])
    points(x=xs[1], y=ys[1], col="red", pch=16)
    arrows(x0=xs[1:19], y0=ys[1:19], x1=xs[2:20], y1=ys[2:20], 
           length=.05, lwd=2, col=rgb(0,0,0,alpha=.5))
  }
}
windows()
  spaghetti.in.2d("gini","total_trade")

enter image description here

I would make these individually, rather than a plot matrix, to have any hope of seeing anything in them. Lastly, I would note that you can't see anything in the plots I've made here because there is, in fact, no structure / information in the data as simulated—I don't know how informative the might be with your real data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.