What's a good way of graphically representing a very large number of paired datapoints?

Question

In my field the usual way to plot paired data is as a series of thin sloping line segments, overlaying it with the median and CI of the median for the two groups:

However, this sort of plot becomes much harder to read as the number of datapoints gets very large (in my case I have on the order of 10000 pairs):

Reducing the alpha helps a bit, but it's still not great. While searching for a solution I came across this paper, and decided to try implementing a 'parallel line plot'. Again, it works very nicely for small numbers of datapoints:

But it's even harder to make this kind of plot look good when the $N$ is very large:

I suppose I could separately show the distributions for the two groups, e.g. with boxplots or violins, and plot a line with errorbars on top showing the two medians/CIs, but I really don't like that idea, since it wouldn't convey the paired nature of the data.

I'm also not overly keen on the idea of a 2D scatter plot: I would prefer a more compact representation, and ideally one in which the values for the two groups are plotted along the same axis. For the sake of completeness, here is what the data looks like as a 2D scatter:

Does anyone know of a better way to represent paired data with a very large sample size? Could you link me to some examples?

Edit

Sorry, I clearly haven't done a good enough job at explaining what I'm looking for. Yes, the 2D scatter plot does work, and there are many ways in which it could be improved in order to convey the density of points better - I could colour-code the dots according to a kernel density estimate, I could make a 2D histogram, I could plot contours on top of the dots etc., etc...

However, I think this is overkill for the message that I'm trying to convey. I don't really care about showing the 2D density of points per se - all I need to do is to show that the values for 'bars' are generally larger than those for 'dots', in as simple and clear a way as possible, and without losing the essential paired nature of the data. Ideally I'd like to plot the paired values for the two groups along the same rather than orthogonal axes, since this makes it easier to visually compare them.

Maybe there is no better option than a scatter plot, but I'd like to know if there are any alternatives that might work.

Answer 1

Given how I understand your aim, I'd just calculate paired differences (bars - dots), then plot these differences in a histogram or kernel density estimate plot. You could also add any combination of (1) a vertical line corresponding to zero difference (2) any choice of percentiles.

This would highlight what portion of the data have bars exceeding dots, and generally what the observed differences are.

(I've assumed that you're not interested in displaying the actual, raw values of bars and dots in the same plot.)

One could also plot confidence or posterior credible intervals to indicate whether these differences are significant. (H/T @MrMeritology!)

Answer 2

With so many pairs you have the possibility of investigating more profoundly the structure, like if the difference $y_B - y_A$ depends on the "starting point" $y_A$!

You could fit a model like $$ y_B=\mu+\text{offset}(y_A) +\Delta (y_A-\bar{y}_A) + \epsilon $$ and you could even add a quadratic term $+\Delta_2 (y_A-\bar{y}_A)^2$ or you could replace the linear+quadratic term with a spline using a generalized additive model (or regression splines).

Graphically you could show the lines as you have shown, with a reduced alpha factor (*), maybe reducing further by only showing a random sample of lines. Then you could color the lines according to slope ...

For Bland-Altman plots, mentioned in a comment by Nick Cox, see for instance for an example Agreement between methods with multiple observations per individual or look through the tag bland-altman-plot.

(*) alpha factor here is a graphical parameter making points in the plot transparent, so the first plotted points is not totally occulted by later overplotting.

Answer 3

I would prefer the 2D scatter plot. I would draw the reference line in light gray for more contrast in the crowded region. To alleviate crowding, draw the markers without border, further reduce alpha, reduce marker size.

That said, if you are more interested in the typical pairs than in the wings of the distribution, try line-plotting the cumulative sum of the dots versus the cumulative sum of the bars. The plot is still 2D but with much less ink. To save also plotting area, you may rotate the trace by 45° so that the frame serves as the reference direction.

That plot would also show any trend in the data. If the process is known to be stationary, sort the pairs by, eg, their geometric mean, sqrt(bars*dots).

Answer 4

I would recommend plotting the lines as you have them for the median and the quartiles, or as many percentiles as you would like for that matter. The median could remain thicker/more discernible than than other percentile lines. This would help preserve the ability to see how the data behave across the distribution without compromising the simplicity and familiarity of the plot that is currently used in your field.

Also, with such a high sample size, the mean or median trend with error bars would likely be sufficient since you would so thoroughly be enjoying the central limit theorem. The biomedical field also relies on those paired line plots, but this is often the case because the sample size could be on the order of 10-20, so it is important to visualise potential leverage points.

Answer 5

My first suggestion is a scatter plot.

If 10000 dots unevenly spread in your plot is still a vague cloud, consider a heat map. The colour of the pixel at x = 10.5, y = 11.5 would indicate how many times value between 10.45 and 10.55 is mapped onto a value between 11.45 and 11.55 : 0 = white = RGB(255,255,255), 1 = blue = RGB(0,0,255), 2 = RGB(1,0,254), ... 256 and above = RGB(255,0,0) = red