I am trying to characterize latency measurements from a microbenchmark; that is, timing of very small segments of computer programs. I'm struggling to identify good descriptors of scale for this data. Also, I would like these calculations to be linear, with respect to the data-size. (No pairwise calculations, like the Mean Absolute Difference (MD), or Med-couple.)
The consumer of the information will be software developer practitioners, who will use the resulting numbers from one experiment to get an understanding of how code behaves, and to compare the results from two experiments, to made decisions about which code behaves "better". (This can mean many things: lower variability, better worst-case behavior, better best-case behavior; sometimes one aspect improves, and another aspect regresses.)
My measurements are not normally distributed, or even shifted log-normal distributed; the systems being measured are pre-asymptotic.
My basic system model is a linear combination of:
- scalar positive offset
- sum of zero or more different positive-scaled Bernoulli distributions
- sum of zero or more different uniform [0,+b] distributions
- sum of zero or more different exponential distributions
- measurement error
- model error
(I'm not looking to estimate the parameters of this model. I describe this only to inform the potential distributions of the data, and what the summary descriptive-statistics need to handle.)
As such, the data is:
- never negative.
- not symmetric, and likely right-skewed.
I currently have:
- Five-number summary: minimum, Q1, median, Q3, maximum (normalized and shifted by the median)
- IQR
- MAD (median, from the median)
The normalized and shifted five-number summary gives a rough idea of the distribution of the values about the median "center". (If I were to graph these values, I'd graph them un-shifted and un-scaled with the time-axis always starting at zero; the spacing between the values on the graph would give the sense of scale/spread.) Overall, I'm content with these values.
The IQR is nice, since it describes the center 50% of the data. Yet, since this is a skewed distribution, the center 50% of the data will not necessarily describe the peak or the densest part of the data. (Consider the case where one exponential distribution dominates the rest of the model factors; the peak is the leftmost value, and the [min,median] contains the densest part of the distribution.)
The MAD seems to be nicer than the IQR, as it (empirically) always measures a smaller distance than the IQR half-range. Yet, it represents a symmetric range about the median, which also seems to ignore the skewedness aspect.
So, both IQR and MAD measure something, but it seems to ignore the skewness of the distribution. Or, IQR and MAD seem like fine measures for symmetric distributions; in those cases, they will approximate each other, which for the Gaussian (error) Distribution will be centered on the peak value (mu), and contain the densest 50% of the distribution.
Given my critique above of the IQR and MAD, I'm considering identifying the narrowest range that contains 50% of the samples. This calculation can be done in linear-time, and has an easy to explain meaning. But, I'm unaware of anyone else using such a measurement, or what it should be called.
So, given my data model with non-negative, non-symmetric (likely right-skewed) values, my requirement for a linear data calculation, and my desire for a meaningful, descriptive statistic for scale, what alternatives should I consider?
Addendum:
Since it may come up in comments/answers, I am using multiple measures of location, since each has a distinct purpose:
- minimum: mostly robust; useful as it's the most "optimistic" answer in the absence of system variability.
- median, with 9X% confidence-intervals: very robust; useful to say 50% of samples are within a certain range of the minimum, in the presence of system noise.
- mean: somewhat fragile; useful for estimating aggregate behaviour over a sequence of repeated operations.
- maximum: very fragile; useful for doing worst-case analysis.
- (I'm not using mode, this seems very fragile to system noise.)
