# Difference between confidence intervals and comparison arrows

I am using the R package lsmeans. On page 10 of the vignette (pdf), a way to show the comparisons between groups is described using:

plot(days.lsm, comparison=TRUE, alpha=0.10)


This displays means together with their confidence intervals (CIs) and comparison arrows. What do the comparison arrows represent? How can the means of 2 groups be statistically different if the CIs overlap?

• Please note - they are CONFIDENCE intervals and COMPARISON arrows. Getting that straight will help answer your question. – Russ Lenth Sep 23 '15 at 1:41
• I edited the post accordingly. – Russ Lenth Sep 23 '15 at 2:01

This seems to be described in the vignette you quote (p. 10, italics added by myself):

Another way to display pairwise comparisons is via the comparisons argument of plot. When this is set to TRUE, arrows are added to the plot, with lengths set so that the amount by which they overlap (or don't overlap) matches as closely as possible to the amounts by which corresponding confidence intervals for differences cover (or don't cover) the value zero. This does not always work, and if there are discrepancies, a message is printed. But it usually works. (...) these comparison arrows show more about the degree of significance in the comparisons. The lowest and highest LS mean have arrows pointing only inward, as the others are not needed.

As you can see, this is rather informal method of comparison. To see how the arrows are obtained you can check the source code of lsmeans package (in R/plot.lsm.R file).

• Hi, thanks for your answer. Yes I read that part, but I did not understand what the arrows represent...the standard error? – gabboshow Sep 21 '15 at 13:15
• Thanks for the edit... any idea how these arrows are calculated? Do you have any suggestion on how to report these results in a scientific paper? I ve never seen these arrows.... – gabboshow Sep 21 '15 at 13:19
• @gabboshow I never seen them used anywhere besides this package, it seems to be rather "informal" way of exploratory analysis and you'd probably have to refer to this vignette. – Tim Sep 21 '15 at 13:28
• considering that 2 means (x1,x2) are sign. different if x1-x2>1.96*sqrt(SE1^2+SE2^2) and the 2 means have not overlapping CI if x1-x2>1.96*(SE1^2+SE2^2) they might represent the standard error that falls within the region where the difference becomes significantly different before the two group confidence intervals cease to overlap. what do you think? – gabboshow Sep 21 '15 at 13:35
• @gabboshow I would not attach too much meaning to it... it is an informal way of comparisons, as described in docs, that "usually works" as authors noticed. – Tim Sep 21 '15 at 13:41

I'll try to explain it briefly.

First, it is important to know that one should never use side-by-side confidence intervals to do comparisons. The reason is because the standard errors of means are not the same as the standard errors of differences of means. In fact, they can be hugely different, as when comparing levels of a within-subjects treatment, where the SEs for means include between-subjects variations but the SEs for differences exclude the between-subjects variations.

So if you want to do graphical comparisons where you assess the degree to which things overlap, you need something other than confidence intervals. The arrows in the lsmeans package are derived from the lengths of the confidence intervals for the differences between pairs of means, and how much these intervals overlap zero. Accordingly, a measure of "covering zero" is computed (I'm skipping the formula, but the more an interval covers zero, the greater it is, and the more it doesn't, the more negative it is). Now, I also define a measure (on the same scale) of how much two opposite-pointing arrows overlap.

The algorithm uses weighted least-squares to fit the lengths of the arrows (as regression parameters) so that the overlap values match the zero coverages. Greater weight is given to cases where the coverage is only slightly above or below zero, as it's most important to get those as accurate as possible.

It is an ad hoc method, but it often works; and an error is reported when the solution results in arrows that overlap when they shouldn't, or vice-versa.

In the simplest case where every pairwise difference has the same standard error, there is a closed-form solution where each arrow's length is 1/4 the length of the confidence interval width. The cases where it doesn't work occur where there are widely varying standard errors.

I have been meaning to write up a little article or vignette on this, but have not gotten around to it yet.