Difference between confidence intervals and comparison arrows I am using the R package lsmeans. On page 10 of the vignette (pdf), a way to show the comparisons between groups is described using:
plot(days.lsm, comparison=TRUE, alpha=0.10)


This displays means together with their confidence intervals (CIs) and comparison arrows. What do the comparison arrows represent? How can the means of 2 groups be statistically different if the CIs overlap?
 A: I'll try to explain it briefly.
First, it is important to know that one should never use side-by-side confidence intervals to do comparisons. The reason is because the standard errors of means are not the same as the standard errors of differences of means. In fact, they can be hugely different, as when comparing levels of a within-subjects treatment, where the SEs for means include between-subjects variations but the SEs for differences exclude the between-subjects variations.
So if you want to do graphical comparisons where you assess the degree to which things overlap, you need something other than confidence intervals. The arrows in the lsmeans package are derived from the lengths of the confidence intervals for the differences between pairs of means, and how much these intervals overlap zero. Accordingly, a measure of "covering zero" is computed (I'm skipping the formula, but the more an interval covers zero, the greater it is, and the more it doesn't, the more negative it is). Now, I also define a measure (on the same scale) of how much two opposite-pointing arrows overlap.
The algorithm uses weighted least-squares to fit the lengths of the arrows (as regression parameters) so that the overlap values match the zero coverages. Greater weight is given to cases where the coverage is only slightly above or below zero, as it's most important to get those as accurate as possible. 
It is an ad hoc method, but it often works; and an error is reported when the solution results in arrows that overlap when they shouldn't, or vice-versa.
In the simplest case where every pairwise difference has the same standard error, there is a closed-form solution where each arrow's length is 1/4 the length of the confidence interval width. The cases where it doesn't work occur where there are widely varying standard errors.
I have been meaning to write up a little article or vignette on this, but have not gotten around to it yet.
A: This seems to be described in the vignette you quote (p. 10, italics added by myself):

Another way to display pairwise comparisons is via the comparisons
  argument of plot.  When this is set to TRUE, arrows are added to the
  plot, with lengths set so that the amount by which they overlap (or
  don't overlap) matches as closely as possible to the amounts by which
  corresponding confidence intervals for differences cover (or don't
  cover) the value zero.  This does not always work, and if there are
  discrepancies, a message is printed.  But it usually works. (...)
  these comparison arrows show more about the degree of significance in
  the comparisons.  The lowest and highest LS mean have arrows pointing
  only inward, as the others are not needed.

As you can see, this is rather informal method of comparison. To see how the arrows are obtained you can check the source code of lsmeans package (in R/plot.lsm.R file).
