11
$\begingroup$

Suppose we're looking at this box-and-whisker plot:

plot

Between Thursday and Friday, I think most would agree there seems to be a significant difference in time slept. Is that a statistically-valid conjecture, though? Can we discern significant differences due to the fact neither of the inner-quartile ranges overlap between Thursday and Friday? What about the fact that the upper and lower whiskers of Thursday and Friday, respectively, overlap? Does that affect our analysis?

Usually accompanying a chart like this would be some sort of ANOVA, but I'm just curious how much we can say about differences between groups simply by looking at a boxplot.

$\endgroup$
  • $\begingroup$ The circles represent outliers. $\endgroup$ – Michael Chernick Feb 17 '17 at 14:08
  • 3
    $\begingroup$ As long as the plot misses any indication of sample size, that is difficult. But if you include with the plot confidence intervals for the medians, you coulld compare those confidence intervals. They do not seem to be present in your plot. $\endgroup$ – kjetil b halvorsen Feb 17 '17 at 14:10
  • $\begingroup$ @kjetilbhalvorsen this is just a plot I grab from Google :) ... I have included, on my own plot, exactly what you've described, as part of a Tukey's HSD test $\endgroup$ – blacksite Feb 17 '17 at 14:20
  • 2
    $\begingroup$ Without CIs, you can't talk about "significant" differences. However, I would say there is a "notable" difference between Thursday and Friday. Or even "the most notable" difference occurs between Thursday and Friday.. $\endgroup$ – Ashe Feb 17 '17 at 14:23
  • 1
    $\begingroup$ The circles are points more than 1.5 IQR from the nearer quartile. They are not outliers unequivocally and objectively. That for Thursday doesn't look extraordinary compared with the rest of the distribution. That for Friday really does; and a researcher or analyst should want to check it out if at all possible and see if there is a story to explain. Perhaps someone really did not sleep! Flagging data points in this way is flagging them for inspection and thought. It's not a statistical method of identifying demons to be exorcised. $\endgroup$ – Nick Cox Feb 17 '17 at 15:31
4
$\begingroup$

No, you can't. If you had the sample sizes and a lot of experience you might be able to guess - and the accuracy of your guess would depend on (in addition to the effect size) the sample size. If N = 1,000,000 per group, lots of significance. If N = 10 per group, not so much. At 100 per group it's harder to guess.

I'd argue that that is a good thing. The thing to do with a box plot is not to try to guess at statistical significance but to look at what's going on and try to reason about it. Hmm. More sleeping on weekends. That's interesting but not really surprising. We could model hours of sleep as a function of weekend vs. not. Or we could try to see if this pattern varied. Maybe retired people don't have this pattern? What about shift workers? People who work on the weekends? People who work 7 days a week?

As my favorite professor in grad school (Herman Friedman) used to say: "Stop p-ing on the research!"

$\endgroup$
  • 1
    $\begingroup$ I think this reply is unnecessarily pessimistic. The boxplots actually contain some information about the group sizes, because really small group sizes ($N \lt 5$) have characteristic "degenerate" structures. Because these distributions aren't too skewed and have few outliers, the IQR (times a suitable multiple) is a good surrogate for the SD, whence we can upper-bound the standard errors. Thus, one can do a conservative rough-and-ready ANOVA from the plot alone: and it will show the overall ANOVA is significant. One can also do conservative post-hoc tests. $\endgroup$ – whuber Mar 4 '17 at 22:45
8
$\begingroup$

Yes, you can. At least in an approximate sense.

I outline how below (and indeed there's a relationship to "box-overlap" as you suggest) along with some caveats and limitations. But first let's discuss a few preliminaries for some background and context. (I think an appropriate answer here should focus not on the particulars of the example - though that perhaps merits some mention as an aside - but on the central issue of using boxplots to assess whether apparent differences can readily be explained away as random variation or not.)

If you have access to the data you can draw notched boxplots which are designed for this sort of visual comparison.

Image showing notched boxplots in R

There's a discussion of notched boxplot calculations here. If the notch-intervals don't overlap the two groups being compared are approximately different at the 5% level; the calculations are based on calculations at the normal, but they're pretty robust and perform reasonably well across a range of distributions. (If it is treated as a formal test the power isn't so high at the normal but it should do pretty well for a variety of more or less "typical" heavier-tailed cases.)

Considering how notched boxplots work you can discern a quick rule of thumb that will work when you only have a display like the one in the question. When the sample size is 10 and the median is placed close to the middle of the box, the notches in a notched boxplot are about the width of the box, so the notch-ends and the box are in roughly the same place.

See here for discussion of how an "$n=10$" rule of thumb arises.

However, you don't need the median in the middle of the box for this comparison; that only explains how we arrived at the rule. Though we started from notched boxplots and a normal-based calculation of an interval for the median, we're now just considering the "box-overlap" rule at $n=10$ and a null that (along with any further assumptions) would result in identical continuous distributions vs some alternative that would tend to separate the boxes (not necessarily pure location shift, though that's the easiest alternative to interpret).

The probabilities of the possible relative orderings of the quartiles (hinges in a boxplot that sticks to Tukey's definition) in samples sizes where they occur at single observations doesn't depend on the distribution shape under the null. In that case (e.g. at $n=9$ in each sample) this version of the test test is distribution free. At $n=10$ it's not distribution free (since the distribution of the averages of adjacent order statistics does now relate to distribution shape) but it's nearly distribution-free.

Type I error rates near $n=10$: Simulation across a number of commonly-used distributions (both symmetric and skew, heavy and light tailed) show that the two-sample box-overlap test has about a 2.3% significance level at $n=10,10$ (there's really not much variation across distributions) and it's about a 5.6% test at $n=9,9$ (it's back below 5% at $n=8,8$, presumably because of the averaging of order statistics reducing the variance more than the loss of an observation increases it). If you have samples of 9 and 10 the significance level is below 5%.

Other sample sizes: If you know the sample sizes you can figure out where the notches go just from the display. If you have a lower bound on the sample sizes, you can get an upper bound on the notch-locations. But even if all you know is that $n$ is at least 10 you can quickly check for box-overlap. The width of the notch-intervals are proportional to $\sqrt{n}$ so you can work out that at $n=40$, the notches should be about half-way to each quartile from the median.

Looking at your plot:

Note that we can tell from the appearance of the plot in the question that the sample sizes must be at least 5; if they were less than 5 the individual-sample boxplots would have distinct clues that they were from a lower sample size (such as medians being dead center of each box, or the whisker being of length 0 when there was an outlier).

Alternatively if the boxes (marking the quartiles) don't overlap each other and the sample size is at least 10, then the two groups being compared should have different medians at the 5% level (considered as a single pairwise comparison).

If you don't know the $n$'s, since we know that the sample sizes should be at least 5, you just need to make the intervals a little larger than the boxes, specifically, if you extend each box about 40% of the distance from the median and they still don't overlap they'd indicate a significant difference for $n=5$ -- returning here to an argument from notched-boxplot reasoning rather than the broader basis we can discern for just comparing the box.

[Note, that this takes no account of the number of comparisons, so if you're doing multiple comparisons your overall type I error will be larger. It's meant for a visual inspection rather than formal testing; nevertheless the ideas involved can be adapted to a more formal approach, including adjustment for multiple comparisons.]

Having addressed whether you can, it would be reasonable to consider whether you should. Perhaps not; the issue of potential p-hacking is real, but if you're using this to figure out whether, for example, to pursue collecting new data on the research issue and all you have is a boxplot in a paper - say - it may be quite useful to be able to make some assessment of whether there's more there than could easily be explained by variation due to noise. But to consider that issue in depth would really be answering a different question.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.