# How to display error bars for cross-over (paired) experiments

The following scenario has become the Most-FAQ in the trio of investigator (I), reviewer/editor (R, not related to CRAN) and me (M) as plot creator. We can assume that (R) is the typical medical big boss reviewer, who only knows that each plot must have error bar, otherwise it is wrong. When a statistical reviewer is involved, problems are much less critical.

Scenario

In a typical pharmacological cross-over study, two drugs A and B are tested for their effect on glucose level. Each patient is tested twice in random order and under the assumption of no carry-over. The primary endpoint is the difference between glucose (B-A), and we assume that a paired t-test is adequate.

(I) wants a plot that shows the absolute glucose levels in both cases. He fears (R)'s desire for error bars, and asks for standard errors in bar graphs. Let's not start the bar graph war here ._)

(I): That cannot be true. The bars overlap, and we have p=0.03? That's not what I have learned in high school.

(M): We have a paired design here. The requested error bars are totally irrelevant, what counts is the SE/CI of the paired differences, which are not shown in the plot. If I had a choice and there were not too many data, I would prefer the following plot

Added 1: This is the parallel coordinate plot mentioned in several responses

(M): The lines show the pairing, and most lines go up, and that's the right impression, because the slope is what counts (ok, this is categorical, but nevertheless).

(I): That picture is confusing. Nobody understands it, and it has no error bars (R is lurking).

(M): We could also add another plot that shows the relevant confidence interval of the difference. The distance from the zero-line gives an impression of the effect size.

(I): Nobody does it

(R): And it wastes precious trees

(M): (As a good German): Yes, point on the trees is taken. But I nevertheless use this (and never get it published) when we have multiple treatments and multiple contrasts.

Any suggestions? The R-Code is below, if you want to create a plot.

# Graphics for Crossover experiments
library(ggplot2)
library(plyr)
theme_set(theme_bw()+theme(panel.margin=grid::unit(0,"lines")))
n = 20
effect = 5
set.seed(4711)
glu0 = rnorm(n,120,30)
glu1 = glu0 + rnorm(n,effect,7)
dt = data.frame(patient = rep(paste0("P",10:(9+n))),
treatment = rep(c("A","B"), each=n),glucose = c(glu0,glu1))

dt1 = ddply(dt,.(treatment), function(x){
data.frame(glucose = mean(x$glucose), se = sqrt(var(x$glucose)/nrow(x)) )})

tt = t.test(glucose~treatment,paired=TRUE,data=dt,conf.int=TRUE)
dt2 = data.frame(diff = -tt$estimate,low=-tt$conf.int[2], up=-tt$conf.int[1]) p = paste("p =",signif(tt$p.value,2))

png(height=300,width=300)
ggplot(dt1, aes(x=treatment, y=glucose, fill=treatment))+
geom_bar(stat="identity")+
geom_errorbar(aes(ymin=glucose-se, ymax=glucose+se),size=1., width=0.3)+
geom_text(aes(1.5,150),label=p,size=6)

ggplot(dt,aes(x=treatment,y=glucose, group=patient))+ylim(0,190)+
geom_line()+geom_point(size=4.5)+
geom_text(aes(1.5,60),label=p,size=6)

ggplot(dt2,aes(x="",y=diff))+
geom_errorbar(aes(ymin=low,ymax=up),size=1.5,width=0.2)+
geom_text(aes(1,-0.8),label=p,size=6)+
ylab("95% CI of difference glucose B-A")+  ylim(-10,10)+
theme(panel.border=element_blank(), panel.grid.major.x=element_blank(),
panel.grid.major.y=element_line(size=1,colour="grey88"))

dev.off()

-
Oh my goodness @dieter-menne, great question! It is so dependent on the type of journal to which you submit the article. I will stay away from the graph wars, but I love graphs 2 and 3: so much information in so little space. –  doug.numbers Jun 3 '13 at 16:43
Masson and Loftus (2003) is an oft given reference. They show example plots in which they panel the CI of the difference in a second panel of the same graph. Whether that will make the everyone in the circus happy I do not know (I feel for you though!) –  Andy W Jun 3 '13 at 17:00
Thanks, @AndyW, for the reference. It is to the point, but has a problem: it is from psychology. There are many more good papers on the subject from psychologists who have a much better statistical background compared to medical reviewers. I wish I could find some high ranking journal's guide on the subject, that's the only sharp sword I could handle. –  Dieter Menne Jun 3 '13 at 18:32

You are totally correct in your assumption that error bars representing the standard error of the mean are totally inappropriate for within-subject designs. However, the question of overlapping error bars and significance is yet another topic, to which I will come back at the end of this commented reference list.

There is rich literature from Psychology on within-subject confidence intervals or error bars which do exactly what you want. The reference work is clearly:

Loftus, G. R., & Masson, M. E. J. (1994). Using confidence intervals in within-subject designs. Psychonomic Bulletin & Review, 1(4), 476–490. doi:10.3758/BF03210951

However, their problem is that they use the same error term for all levels of a within-subject factor. This does not seem to be a huge problem for your case (2 levels). But there are more modern approaches solving this problem. Most notably:

Franz, V., & Loftus, G. (2012). Standard errors and confidence intervals in within-subjects designs: Generalizing Loftus and Masson (1994) and avoiding the biases of alternative accounts. Psychonomic Bulletin & Review, 1–10. doi:10.3758/s13423-012-0230-1

Baguley, T. (2011). Calculating and graphing within-subject confidence intervals for ANOVA. Behavior Research Methods. doi:10.3758/s13428-011-0123-7 [can be found here]

Further references can be found in the latter two papers (which I think are both worth a read).

How do researchers interpret CIs? Bad according to the following paper:

Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005). Researchers Misunderstand Confidence Intervals and Standard Error Bars. Psychological Methods, 10(4), 389–396. doi:10.1037/1082-989X.10.4.389

How should we interpret overlapping and non-overlapping CIs?

Cumming, G., & Finch, S. (2005). Inference by Eye: Confidence Intervals and How to Read Pictures of Data. American Psychologist, 60(2), 170–180. doi:10.1037/0003-066X.60.2.170

One final thought (although this is not relevant to your case): If you have a split-plot design (i.e., within- and between-subject factors) in one plot, you can forget about error bars all together. I would (humbly) recommend my raw.means.plot function in the R package plotrix.

-
Very useful reference list. Cumming has brought many of his ideas together in amazon.com/Understanding-New-Statistics-Meta-Analysis-ebook/dp/… (Whatever you want to say about the expression "the new statistics", I probably agree.) –  Nick Cox Jun 4 '13 at 13:55

The question does not seem to be about error bars so much as about the best ways of plotting paired data.

In essence error bars here are at most a way of summarizing uncertainty: they do not, and they necessarily cannot, say much about any fine structure in the data.

Parallel coordinate plots -- sometimes called profile plots, a term that means different things in different fields -- have been mentioned in the question, Basic scatter plots have already been suggested by @Ray Koopman.

A specialized scatter plot popular here and there is a plot of difference (here $A - B$, say) versus mean (or sum) $(A + B)/2$ or $A + B$. In medicine this is often known as a Bland-Altman plot (perhaps because it was earlier used by Oldham) and in statistics it is often known as a Tukey mean-difference plot.

Another source for this plot is Neyman, J., Scott, E. L. and Shane, C. D. 1953. On the spatial distribution of galaxies: a speciﬁc model. Astrophysical Journal 117: 92–133.

In broad terms such plots resemble the idea of plotting residuals versus fitted, also popularised by Tukey and his brother-in-law-squared Anscombe.

The key idea of such plots is that the horizontal line of no difference $A - B = 0$ is naturally equivalent to the line of equality $A = B$, but it is often easier psychologically to work with a horizontal reference line. In addition, if $A$ and $B$ are broadly similar, a scatter plot uses much of its space in emphasising that fact, whereas the structure of the differences should be of greater interest.

A neglected design is the parallel-line plot of McNeil, D.R. 1992. On graphing paired data. American Statistician 46: 307–310. This is also discussed in the two references below.

Stata-linked reviews, with several references, are in

2004, Graphing agreement and disagreement. Stata Journal 4: 329-349.

.pdf accessible at http://www.stata-journal.com/sjpdf.html?articlenum=gr0005

Paired, parallel, or profile plots for changes, correlations, and other comparisons. Stata Journal 9: 621-639.

.pdf accessible at http://www.stata-journal.com/sjpdf.html?articlenum=gr0041

Non-Stata users should able to skip and hum their way through the Stata code while working out how to implement the graphs in their own favourite software.

In all cases if the ratio of $A$ and $B$ is of interest rather than its difference, exactly the same ideas should be used, but employing logarithmic scales.

-

Try a scatter plot of the individual (A,B) points. Most of them should lie on only one side of the diagonal (the line A = B). There are two analogs of error bars. The conventional one, equivalent to a CI for the mean difference, would be a confidence band for the mean difference. The band would be the region between two lines, both of which are parallel to the diagonal. A paired t-test would be significant if and only if both edges of the band are on the same side of the diagonal.

A more conservative error-bar analog would be a confidence ellipse for the centroid.

-
@dieter-menne: Would clinical chemistry be medical enough? You should be able to find a lot of $c_{method B}$ over $c_{method A}$ plots there. Paper is 2d, paired data is 2d, sounds like a nicely fitting situation to me... That would even allow for a further (symbol shape or color) dimension indicating who got treatment A first and who started with treatment B (so you can check whether there is really no difference between those two groups) –  cbeleites Jun 3 '13 at 21:34
No, cl. -:) Clinical chemistry is too sophisticated. Could it be that you had added some reference in "there"; I am not sure what paper you are speaking of. –  Dieter Menne Jun 4 '13 at 12:11

Preliminary summary:

Masson/Loftus is very exhaustive, and not an easy reading to give to my medical colleagues who would not accept something like an "interaction". They also have some suggestions for multiple comparisons, which show that pairwise confidence intervals are difficult to illustrate when one does not want to simplify heavily.

I don't like this style: the bars with error bars look last millennium Excelish. However, they also use a slightly more elegant style:

Cumming/Finch and Belia et al. are must readings. The first is the perfect choice to give your friend who shudders when (s)he sees the word interaction. I ordered Cumming's book after reading that article. The second shows a test I will implement in Shiny for the next medical investigator meeting.

I like this plot, even if there is a second axis which I never used before; check Henrik's and some other's contribution on StackOverflow for an R-base graphics method to obtain it. I would prefer to put the second axis to the left of the difference to make absolutely clear that the values changed, and maybe add a p-value axis.

Anybody from the lattice/ggplot fraction taking a shot? All supplied solutions are base graphics and not panelizable/facetable.

However: note that comments and papers are mostly from the psychology department (and @cbeleites from hardcore chemistry). It would good to get comments from reviewers of medical journals.

-

Why not just plot the difference* for each patient? You could then use a histogram, a box plot or a normal probability plot and overlay a 95% confidence interval for the difference.

• In some scenarios it might be the difference of the logarithms. See, for example, Patterson & Jones, "Bioequivalence and Statistics in Clinical Pharmacology", Chapman, 2006.
-
Why not? It might provide valid information, but plots with individual points or pairs are not really popular in medical research. That's why parallel plot (which provide more information) are so unpopular. Medical researchers want averages and "normal ranges" (whatever that is). –  Dieter Menne Jun 4 '13 at 21:46