35
$\begingroup$

I'd like to know if there is a boxplot variant adapted to Poisson distributed data (or possibly other distributions)?

With a Gaussian distribution, whiskers placed at L = Q1 - 1.5 IQR and U = Q3 + 1.5 IQR, the boxplot has the property that there will be roughly as many low outliers (points below L) as there are high outliers (points above U).

If the data is Poisson distributed however, this does not hold anymore because of the positive skewness we get Pr(X<L) < Pr(X>U). Is there an alternate way to place the whiskers such that it would 'fit' a Poisson distribution?

$\endgroup$
5
  • 2
    $\begingroup$ Try logging it first? You might also say what you want your boxplot to be 'well adapted' to. $\endgroup$ Commented Jul 15, 2011 at 11:24
  • 2
    $\begingroup$ There is one problem with doing such modification -- people are used to standard boxplot definition and will most likely assume it when looking at the plot whether you like it or not. Thus, this may bring more confusion than gain. $\endgroup$
    – user88
    Commented Jul 15, 2011 at 15:33
  • $\begingroup$ @mbq:> the thing with boxplots is they combine two features unto one tool; a data visualization feature (the box) and an outlier-detection feature (the whiskers). What you say is absolutly true of the former, but the later could use a skew adjustment. $\endgroup$
    – user603
    Commented Jul 25, 2011 at 9:12
  • $\begingroup$ @conjugateprior Here's a Poisson sample: 0, 0, 1, 0, 1, 2, 0, 0, 1, 0, 0 .... notice a problem with just taking logs? $\endgroup$
    – Glen_b
    Commented Feb 25, 2013 at 2:13
  • $\begingroup$ @Glen_b That must be why it's a comment not an answer. And why it has two parts. $\endgroup$ Commented Feb 25, 2013 at 9:18

2 Answers 2

33
$\begingroup$

Boxplots weren't designed to assure low probability of exceeding the ends of the whiskers in all cases: they are intended, and usually used, as simple graphical characterizations of the bulk of a dataset. As such, they are fine even when the data have very skewed distributions (although they might not reveal quite as much information as they do about approximately unskewed distributions).

When boxplots become skewed, as they will with a Poisson distribution, the next step is to re-express the underlying variable (with a monotonic, increasing transformation) and redraw the boxplots. Because the variance of a Poisson distribution is proportional to its mean, a good transformation to use is the square root.

Each boxplot depicts 50 iid draws from a Poisson distribution with given intensity (from 1 through 10, with two trials for each intensity). Notice that the skewness tends to be low.

side-by-side boxplots

The same data on a square root scale tend to have boxplots that are slightly more symmetric and (except for the lowest intensity) have approximately equal IQRs regardless of intensity).

boxplots of transformed data

In sum, don't change the boxplot algorithm: re-express the data instead.


Incidentally, the relevant chances to be computing are these: what is the chance that an independent normal variate $X$ will exceed the upper(lower) fence $U$($L$) as estimated from $n$ independent draws from the same distribution? This accounts for the fact that the fences in a boxplot are not computed from the underlying distribution but are estimated from the data. In most cases, the chances are much greater than 1%! For instance, here (based on 10,000 Monte-Carlo trials) is a histogram of the log (base 10) chances for the case $n=9$:

histogram of chances

(Because the normal distribution is symmetric, this histogram applies to both fences.) The logarithm of 1%/2 is about -2.3. Clearly, most of the time the probability is greater than this. About 16% of the time it exceeds 10%!

It turns out (I won't clutter this reply with the details) that the distributions of these chances are comparable to the normal case (for small $n$) even for Poisson distributions of intensity as low as 1, which is pretty skewed. The main difference is that it's usually less likely to find a low outlier and a little more likely to find a high outlier.

$\endgroup$
2
31
$\begingroup$

There is a generalization of standard box-plots that I know of in which the lengths of the whiskers are adjusted to account for skewed data. The details are better explained in a very clear & concise white paper (Vandervieren, E., Hubert, M. (2004) "An adjusted boxplot for skewed distributions", see here).

There is an $\verb+R+$ implementation of this ($\verb+robustbase::adjbox()+$) as well as a matlab one (in a library called $\verb+libra+$).

I personally find it a better alternative to data transformation (though it is also based on an ad-hoc rule, see white paper).

Incidentally, I find I have something to add to whuber's example here. To the extend that we're discussing the whiskers' behaviour, we really should also consider what happens when considering contaminated data:

library(robustbase)
A0 <- rnorm(100)
A1 <- runif(20, -4.1, -4)
A2 <- runif(20,  4,    4.1)
B1 <- exp(c(A0, A1[1:10], A2[1:10]))
boxplot(sqrt(B1), col="red", main="un-adjusted boxplot of square root of data")
adjbox(      B1,  col="red", main="adjusted boxplot of data")

In this contamination model, B1 has essentially a log-normal distribution save for 20 percent of the data that are half left, half right outliers (the break down point of adjbox is the same as that of regular boxplots, i.e. it assumes that at most 25 percent of the data can be bad).

The graphs depict the classical boxplots of the transformed data (using the square root transformation)

classical boxplot on square root transform of the data

and the adjusted boxplot of the non-transformed data.

adjusted boxplot of un-transformed data

Compared to adjusted boxplots, the former option masks the real outliers and labels good data as outliers. In general, it will contrive to hide any evidence of asymmetry in the data by classifying offending points as outliers.

In this example, the approach of using the standard boxplot on the square root of the data finds 13 outliers (all on the right), whereas the adjusted boxplot finds 10 right and 14 left outliers.

EDIT: adjusted box plots in a nutshell.

In 'classical' boxplots the whiskers are placed at:

$Q_1$-1.5*IQR and $Q_3$+1.5*IQR

where IQR is the inter-quantile range, $Q_1$ is the 25th percentile and $Q_3$ is the 75th percentile of the data. The rule of thumb is to regard everything outside the fence as dubious data (the fence is the interval between the two whiskers).

This rule of thumb is ad-hoc: the justification is that if the uncontaminated part of the data is approximately Gaussian, then less than 1% of the good data would be classified as bad using this rule.

A weakness of this fence-rule, as pointed out by the OP, is that the length of the two whiskers are identical, meaning the fence-rule only makes sense if the uncontaminated part of the data has a symmetric distribution.

A popular approach is to preserve the fence-rule and to adapt the data. The idea is to transform the data using some skew correcting monotonous transformation (square root or log or more generally box-cox transforms). This is somewhat messy approach: it relies on circular logic (the transformation should be chosen so as to correct the skewness of the uncontaminated part of the data, which is at this stage an un-observable) and tends to make the data harder to interpret visually. At any rate, this remains a strange procedure whereby one changes the data to preserve what is after all an ad-hoc rule.

An alternative is to leave the data untouched and change the whisker rule. The adjusted boxplot allows the length of each whisker to vary according to an index measuring the skewness of the uncontaminated part of the data:

$Q_1$-$\exp(M,\alpha)$1.5*IQR and $Q_3$+$\exp(M,\beta)$1.5*IQR

Where $M$ is an index of skewness of the uncontaminated part of the data (i.e., just as the median is a measure of location for the uncontaminated part of the data or the MAD a measure of spread for the uncontaminated part of the data) and $\alpha$ $\beta$ are numbers chosen such that for uncontaminated skewed distributions the probability of lying outside the fence is relatively small across a large collection of skewed distributions (this is the ad-hoc part of the fence rule).

For cases when the good part of the data is symmetric, $M\approx 0$ and we're back to the classical whiskers.

The authors suggest using the med-couple as an estimator of $M$ (see reference inside the white paper) because of its high efficiency (though in principle any robust skew index could be used). With this choice of $M$, they then calculated the optimal $\alpha$ and $\beta$ empirically (using a large number of skewed distributions) as:

$Q_1$-$\exp(-4M)$1.5*IQR and $Q_3$+$\exp(3M)$1.5*IQR, if $M\geq 0$

$Q_1$-$\exp(-3M)$1.5*IQR and $Q_3$+$\exp(4M)$1.5*IQR, if $M<0$

$\endgroup$
7
  • 3
    $\begingroup$ I would be interested to know how you find my example "unhelpful"--just branding it as such is not constructive. I will admit that the example is somewhat disappointing in the sense that the data transformation does not represent a spectacular improvement. That's the fault of the Poisson distributions: they just aren't skewed enough to be worth the bother of all this analysis! $\endgroup$
    – whuber
    Commented Jul 25, 2011 at 14:09
  • 3
    $\begingroup$ @whuber:> first, sorry for the tone: it was from an un-edited first draft and it has been corrected (i typically write shorthand paragraphs meant as note to self, then repeatedly go over them -- this one got lost in the long inter-winded response). Now for the critic itself: your example depicts the behavior of the solution using transformation in the case of uncontaminated data. IMHO the whisker rule should -perhaps preliminary- be evaluated with a contamination model in mind. $\endgroup$
    – user603
    Commented Jul 25, 2011 at 14:30
  • 1
    $\begingroup$ @user Thanks for the clarification. I don't mind the criticism, which is interesting, and I appreciate the references to adjusted boxplots. (+1) $\endgroup$
    – whuber
    Commented Jul 25, 2011 at 14:39
  • $\begingroup$ I agree with user603 that there is a difference in whether you inspect a pure distribution (such as in whubers answer) or have data from a distribution plus some outliers (discussed here as contamination). From my perspective, in real settings, a boxplot is used to scan for outliers. Followingly, an analysis of boxplots that omits outliers somehow misses the point. Therefore, this answer seems to better serve the purpose for using boxplots. $\endgroup$
    – Henrik
    Commented Jul 25, 2011 at 14:48
  • 3
    $\begingroup$ @Henrik Identifying outliers is only one of the many purposes of boxplots. Tukey's approach was first to find an appropriate re-expression of the data that made the middle of their distribution approximately symmetric. This obviates the need for any adjustment for skewness. That already accomplishes a lot in terms of permitting comparisons among boxplots, which is where they become truly useful. "Adjusting" the whiskers completely misses out on this fundamental issue. Therefore I would be wary of using the adjustment: its need is a signal that the analysis is not being done well. $\endgroup$
    – whuber
    Commented Jul 25, 2011 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.