How to display error bars for cross-over (paired) experiments

Question

The following scenario has become the Most-FAQ in the trio of investigator (I), reviewer/editor (R, not related to CRAN) and me (M) as plot creator. We can assume that (R) is the typical medical big boss reviewer, who only knows that each plot must have error bar, otherwise it is wrong. When a statistical reviewer is involved, problems are much less critical.

Scenario

In a typical pharmacological cross-over study, two drugs A and B are tested for their effect on glucose level. Each patient is tested twice in random order and under the assumption of no carry-over. The primary endpoint is the difference between glucose (B-A), and we assume that a paired t-test is adequate.

(I) wants a plot that shows the absolute glucose levels in both cases. He fears (R)'s desire for error bars, and asks for standard errors in bar graphs. Let's not start the bar graph war here ._)

(I): That cannot be true. The bars overlap, and we have p=0.03? That's not what I have learned in high school.

(M): We have a paired design here. The requested error bars are totally irrelevant, what counts is the SE/CI of the paired differences, which are not shown in the plot. If I had a choice and there were not too many data, I would prefer the following plot

Added 1: This is the parallel coordinate plot mentioned in several responses

(M): The lines show the pairing, and most lines go up, and that's the right impression, because the slope is what counts (ok, this is categorical, but nevertheless).

(I): That picture is confusing. Nobody understands it, and it has no error bars (R is lurking).

(M): We could also add another plot that shows the relevant confidence interval of the difference. The distance from the zero-line gives an impression of the effect size.

(I): Nobody does it

(R): And it wastes precious trees

(M): (As a good German): Yes, point on the trees is taken. But I nevertheless use this (and never get it published) when we have multiple treatments and multiple contrasts.

Any suggestions? The R-Code is below, if you want to create a plot.

# Graphics for Crossover experiments
library(ggplot2)
library(plyr)
theme_set(theme_bw()+theme(panel.margin=grid::unit(0,"lines")))
n = 20
effect = 5 
set.seed(4711)
glu0 = rnorm(n,120,30)
glu1 = glu0 + rnorm(n,effect,7)
dt = data.frame(patient = rep(paste0("P",10:(9+n))),              
                treatment = rep(c("A","B"), each=n),glucose = c(glu0,glu1))

dt1 = ddply(dt,.(treatment), function(x){
  data.frame(glucose = mean(x$glucose), se = sqrt(var(x$glucose)/nrow(x)) )})

tt = t.test(glucose~treatment,paired=TRUE,data=dt,conf.int=TRUE)
dt2 = data.frame(diff = -tt$estimate,low=-tt$conf.int[2], up=-tt$conf.int[1])
p = paste("p =",signif(tt$p.value,2))

png(height=300,width=300)
ggplot(dt1, aes(x=treatment, y=glucose, fill=treatment))+      
  geom_bar(stat="identity")+  
  geom_errorbar(aes(ymin=glucose-se, ymax=glucose+se),size=1., width=0.3)+
  geom_text(aes(1.5,150),label=p,size=6)

ggplot(dt,aes(x=treatment,y=glucose, group=patient))+ylim(0,190)+
  geom_line()+geom_point(size=4.5)+
  geom_text(aes(1.5,60),label=p,size=6)

ggplot(dt2,aes(x="",y=diff))+
  geom_errorbar(aes(ymin=low,ymax=up),size=1.5,width=0.2)+ 
  geom_text(aes(1,-0.8),label=p,size=6)+
  ylab("95% CI of difference glucose B-A")+  ylim(-10,10)+
  theme(panel.border=element_blank(), panel.grid.major.x=element_blank(),
         panel.grid.major.y=element_line(size=1,colour="grey88"))

dev.off()

Answer 1

You are totally correct in your assumption that error bars representing the standard error of the mean are totally inappropriate for within-subject designs. However, the question of overlapping error bars and significance is yet another topic, to which I will come back at the end of this commented reference list.

There is rich literature from Psychology on within-subject confidence intervals or error bars which do exactly what you want. The reference work is clearly:

Loftus, G. R., & Masson, M. E. J. (1994). Using confidence intervals in within-subject designs. Psychonomic Bulletin & Review, 1(4), 476–490. doi:10.3758/BF03210951

However, their problem is that they use the same error term for all levels of a within-subject factor. This does not seem to be a huge problem for your case (2 levels). But there are more modern approaches solving this problem. Most notably:

Franz, V., & Loftus, G. (2012). Standard errors and confidence intervals in within-subjects designs: Generalizing Loftus and Masson (1994) and avoiding the biases of alternative accounts. Psychonomic Bulletin & Review, 1–10. doi:10.3758/s13423-012-0230-1

Baguley, T. (2011). Calculating and graphing within-subject confidence intervals for ANOVA. Behavior Research Methods. doi:10.3758/s13428-011-0123-7 [can be found here]

Further references can be found in the latter two papers (which I think are both worth a read).

---

How do researchers interpret CIs? Bad according to the following paper:

Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005). Researchers Misunderstand Confidence Intervals and Standard Error Bars. Psychological Methods, 10(4), 389–396. doi:10.1037/1082-989X.10.4.389

How should we interpret overlapping and non-overlapping CIs?

Cumming, G., & Finch, S. (2005). Inference by Eye: Confidence Intervals and How to Read Pictures of Data. American Psychologist, 60(2), 170–180. doi:10.1037/0003-066X.60.2.170

---

One final thought (although this is not relevant to your case): If you have a split-plot design (i.e., within- and between-subject factors) in one plot, you can forget about error bars all together. I would (humbly) recommend my raw.means.plot function in the R package plotrix.

Answer 2

The question does not seem to be about error bars so much as about the best ways of plotting paired data.

In essence error bars here are at most a way of summarizing uncertainty: they do not, and they necessarily cannot, say much about any fine structure in the data.

Parallel coordinate plots -- sometimes called profile plots, a term that means different things in different fields -- have been mentioned in the question. Basic scatter plots have already been suggested by @Ray Koopman.

A specialized scatter plot popular here and there is a plot of difference (here $A - B$, say) versus mean (or sum) $(A + B)/2$ or $A + B$. In medicine this is often known as a bland-altman-plot (perhaps because it was earlier used by Oldham) and in statistics it is often known as a Tukey mean-difference plot.

Another source for this plot is Neyman, J., Scott, E. L. and Shane, C. D. 1953. On the spatial distribution of galaxies: a specific model. Astrophysical Journal 117: 92–133.

In broad terms such plots resemble the idea of plotting residuals versus fitted, also popularised by Tukey and his brother-in-law-squared Anscombe.

The key idea of such plots is that the horizontal line of no difference $A - B = 0$ is naturally equivalent to the line of equality $A = B$, but it is often easier psychologically to work with a horizontal reference line. In addition, if $A$ and $B$ are broadly similar, a scatter plot uses much of its space in emphasising that fact, whereas the structure of the differences should be of greater interest.

A neglected design is the parallel-line plot of McNeil, D.R. 1992. On graphing paired data. American Statistician 46: 307–310. This is also discussed in the two references below.

Stata-linked reviews, with several references, are in

2004, Graphing agreement and disagreement. Stata Journal 4: 329-349.

.pdf accessible at http://www.stata-journal.com/sjpdf.html?articlenum=gr0005

Paired, parallel, or profile plots for changes, correlations, and other comparisons. Stata Journal 9: 621-639.

.pdf accessible at http://www.stata-journal.com/sjpdf.html?articlenum=gr0041

Non-Stata users should able to skip and hum their way through the Stata code while working out how to implement the graphs in their own favourite software.

In all cases if the ratio of $A$ and $B$ is of interest rather than its difference, exactly the same ideas should be used, but employing logarithmic scales.

Answer 3

Try a scatter plot of the individual (A,B) points. Most of them should lie on only one side of the diagonal (the line A = B). There are two analogs of error bars. The conventional one, equivalent to a CI for the mean difference, would be a confidence band for the mean difference. The band would be the region between two lines, both of which are parallel to the diagonal. A paired t-test would be significant if and only if both edges of the band are on the same side of the diagonal.

A more conservative error-bar analog would be a confidence ellipse for the centroid.

Answer 4

Preliminary summary:

Masson/Loftus is very exhaustive, and not an easy reading to give to my medical colleagues who would not accept something like an "interaction". They also have some suggestions for multiple comparisons, which show that pairwise confidence intervals are difficult to illustrate when one does not want to simplify heavily.

I don't like this style: the bars with error bars look last millennium Excelish. However, they also use a slightly more elegant style:

Cumming/Finch and Belia et al. are must readings. The first is the perfect choice to give your friend who shudders when (s)he sees the word interaction. I ordered Cumming's book after reading that article. The second shows a test I will implement in Shiny for the next medical investigator meeting.

I like this plot, even if there is a second axis which I never used before; check Henrik's and some other's contribution on StackOverflow for an R-base graphics method to obtain it. I would prefer to put the second axis to the left of the difference to make absolutely clear that the values changed, and maybe add a p-value axis.

Anybody from the lattice/ggplot fraction taking a shot? All supplied solutions are base graphics and not panelizable/facetable.

However: note that comments and papers are mostly from the psychology department (and @cbeleites from hardcore chemistry). It would good to get comments from reviewers of medical journals.

Answer 5

Why not just plot the difference* for each patient? You could then use a histogram, a box plot or a normal probability plot and overlay a 95% confidence interval for the difference.

• In some scenarios it might be the difference of the logarithms. See, for example, Patterson & Jones, "Bioequivalence and Statistics in Clinical Pharmacology", Chapman, 2006.