# How would one describe such irregular data?

The situation is as follows (physics based): I have an array (7) of pixel sensors (imagine phone cameras) and a ton (millions) of particles crossing them (very large N). Each particle crossing a sensor will produce a specific kind of "event" (a cluster of pixels that responds to the passage of particles), based on the physics of charge transport inside the sensor. The cluster of pixels has, of course, an integer number. Take a few such particle crossing. For each such event, one would get for the 7 sensors the following cluster size:

Sensor                   0    1    2    3    4    5    6    7
Cluster size event 1:    4    4    6    6    6    6    2    5
Cluster size event 2:    6    4    6    6    4    6    4    5
Cluster size event 3:    6    5    6    6    4    5    4    4


Here are the distributions of sensors 0 and 1, as well as for the rest, overlayed As you can see, they are rather similar, since the sensors are operated at similar settings (differences from manufacturing or electrical components).

My question is: How can I describe such data, such that in the end I can quote an "average" cluster size per sensor (including the uncertainty). And finally, per such set of settings of the sensors, I want to quote an "average" number +- uncertainty.

Note: There is no underlying distribution that describes this data (there is probably some Landau-like charge distribution from, which the cluster is produced, but this is truncated, since the response of the sensor is digital: if the amount of charge per pixel is bigger or smaller than a set threshold, we get a "yes" or "no" binary response).

What I have tried:

1. Mean, RMS: In my opinion they do not make sense, as I do not have a normal behaving distribution.
2. Median ? But how can I calculate the error on the median?!? (using the $$\frac{\sigma\sqrt(pi/2)}{\sqrt(n)}$$ from the asymptotic variance implies a normal distribution).
3. Boxplots: I have produced them, see below: The pink line is the median.

Maybe looking at the median, one would say to just use this, but look at other settings :) Here, you see just how much they can differ...

Any hints would be greatly appreciated! Cheers!

• just out of curiousity. what are the colored boxes just right above density plots in the facets?
– Derf
Commented Feb 12 at 11:41
• @Derf, it is the legend. Commented Feb 12 at 11:44
• It is a redacted legend :) sorry for the confusion. I edited the picture and removed the box
– nyw
Commented Feb 12 at 12:25
• The medians appear to be the same, but the distributions aren't. Therefore don't use median to report differences, because it won't help. Detail: Is there some reason why 5 is often less frequent than both 4 and 6? Commented Feb 12 at 13:46
• There is a reason and it has to do with the geometry of the charge diffusion physics inside a sensor. More concretely, imagine I have such a particle that hits the sensor somewhere within a pixel. While traveling through the sensor, it will create some charges that we measure, but that spread (!). In a first, very basic approximation, the charges go isotropically from the point they are created. It is easier to have even number of pixels that have a charge above a threshold, than it would be to have odd number. It's purely geometrical in a first order approximation.
– nyw
Commented Feb 12 at 14:27

How can I describe such data, such that in the end I can quote an "average" cluster size per sensor (including the uncertainty). And finally, per such set of settings of the sensors, I want to quote an "average" number +- uncertainty.

This depends entirely on what you want to describe. You write that you think mean and RMS doesn't make sense because the distributions are not normal. But this is not correct. The mean exists for skewed data as does RMS. It's true that it's often said that it's not good to use the mean for such data and it's true that the mean is strongly affected by outliers, but ... do you care? I don't know. Maybe you want a measure that's affected by outliers.

If you want to use the median, you could also list median absolute deviation (MAD) or interquartile range (I prefer to show this as two numbers). The median is not affected by outliers (unless there are so many that they hardly qualify as outliers) and has a very high breakdown point.

Or maybe you want the trimmed mean -- these are compromises between the median and mean.

Or, possibly, there is no single number that does what you want and you have to use a five-number or seven-number summary (these aren't used much any more, but Tukey liked them and Tukey knew what he was doing).

Or maybe you want boxplots. However, if you do use boxplots on data with such large N, then I suggest adding jitter to the outliers and making the symbol transparent or very small, so that overplotting is avoided to the extent possible.

• Hello Peter and thank you for your answer! I am indeed not interested in the few outliers in this case, but in describing the general trend. The ultimate goal (if you would refer to the last picture from my post, with the different settings) is to do classification (probably based on some ML algorithm) which, given a track that produces on average in all sensors a specific cluster size, is classified into belonging with a higher probability to the eg: green or red or blue distributions. In order to understand how to make such a model, I wanted first to understand my data.
– nyw
Commented Feb 12 at 12:56
• Would you mind detailing a bit your second paragraph about the MAD and IQR for this specific example? What do you mean exactly by showing as two numbers? Taking the MAD of the set as 1/n SUM |x_i - m(X)|, m(X) would in this case be the median, correct? So, I would calculate this quantity and have median +- MAD ? For the IQR, what should I calculate instead? Finally, when you say the median has a very high breakdown point, what does this mean? Thank you!
– nyw
Commented Feb 12 at 13:01
• I mean to post the IQR as two numbers (the first and third quartile) rather than the difference between them. For defn of breakdown point see en.wikipedia.org/wiki/Robust_statistics Commented Feb 12 at 13:14