Visualizing counts that have some outliers

I am running a project which compares citation counts across datasets per year, and then wants to plot the general change in the frequency in which a given source is cited. Here are two example datasets:

1999 - SOURCE A

{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 23, 70, 0, 0, 0, 0, 0, 0, 0, 32, 0,
0, 0, 25, 0, 3, 13, 0, 1, 0, 0, 0, 0, 11, 0, 0, 0, 0, 0, 0, 0, 0, 8,
0, 0, 6, 0, 0, 0, 1, 0, 22, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 2, 0, 2, 0,
21, 6, 10, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0}


2000 - SOURCE A

{0, 0, 0, 0, 0, 4, 400, 82, 0, 0, 2, 0, 0, 0, 2, 6, 0, 0, 12, 24, 0,
0, 0, 0, 0, 0, 4, 4, 0, 0, 0, 0, 142, 0, 0, 0, 0, 2, 0, 0, 0, 2, 2, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 0, 0, 10, 0, 4, 0, 0, 0}


In this case, when I plot the difference between 1999 and 2000 in a simple line-chart, 2000 seems so much bigger thanks to the data points that have 400, 142.

Is there a good way to visualize this kind of data that stays true to outliers, but still shows the general trend that's going on (if any)?

• Length of both list do not much (83 vs 69) elements. – Piotr Migdal Apr 20 '12 at 15:07
• Just as a note: no matter what visualization you use, these won't end up looking like 'typical' data, b/c counts are distributed as a Poisson, $\mathcal{P}(\lambda)$, not as a Gaussian, $\mathcal{N}(\mu,\sigma^2)$. In addition, you have zero-inflation & overdispersion. If you're not familiar w/ these issues, this provides a quick overview using SAS. – gung - Reinstate Monica Apr 20 '12 at 16:37

To show both a general change and account for all of your data points you could use a simple line chart (for your annual difference) combined with a dot-plot for your data points. Here's an example using the difference between average citations between years. If you're using a different measure for annual difference, you can insert that in the average's place. This chart is pretty easy to do in Excel, just create your line chart for your differences, add the addtional data sets for your scatter chart (XY chart type), then format to suit.

• gung made an excellent point about the distribution of your data. You'll need to make a determination of the best way to compare annual results, then you can plug that value into a chart like this. Average is probably a really poor way to do it (but it was quick and easy for the example chart). – dav Apr 20 '12 at 17:10

Without looking at the data first one could have thought about a boxplot. It gives you immediately information on the central tendency (median) and the dispersion (interquartile range). These two measures are quite robust to outliers. In addition to this, a boxplot also allows to identify outliers.

However, the boxplot will not work here, because of the excess zeroes in your data. The 25th, 50th and 75th percentiles are equal to zero and thus you will not see any box.

An alternative would be a quantile-quantile plot. This is a visual technique to compare the distribution of one variable against the distribution of another variable. In your case the graph will look as follows. You will see immediately that there are three outliers in the 2000 data. But otherwise, the data has a similar distribution in 1999 and 2000. As a supplement, you could also redo the plot without these three points. Then you will be able to see what is going on elsewhere in the distribution.

Try using log scale. As the counts start from zero, you need in fact to plot $\log(1+x)$ or $\text{arsinh}(x)$ (i.e. $\sinh$ inverse); both are logarithms for large $x$, but also map $0$ to $0$).

Square root scales are often used for counts. You don't need then the fudge of log(1 + count) or to resort to an inverse hyperbolic function, likely to seem obscure or mysterious even to audiences who have heard of such a beast.

A downside is that square roots don't pull in the tail so much.

Alternatively, I would have thought about separating out

so many counts of zero (perhaps even shown as text)

and

the non-zero counts, shown graphically. If you did that, you could use log scales for the graphs.