3
$\begingroup$

I am trying to characterize latency measurements from a microbenchmark; that is, timing of very small segments of computer programs. I'm struggling to identify good descriptors of scale for this data. Also, I would like these calculations to be linear, with respect to the data-size. (No pairwise calculations, like the Mean Absolute Difference (MD), or Med-couple.)

The consumer of the information will be software developer practitioners, who will use the resulting numbers from one experiment to get an understanding of how code behaves, and to compare the results from two experiments, to made decisions about which code behaves "better". (This can mean many things: lower variability, better worst-case behavior, better best-case behavior; sometimes one aspect improves, and another aspect regresses.)

My measurements are not normally distributed, or even shifted log-normal distributed; the systems being measured are pre-asymptotic.

My basic system model is a linear combination of:

  • scalar positive offset
  • sum of zero or more different positive-scaled Bernoulli distributions
  • sum of zero or more different uniform [0,+b] distributions
  • sum of zero or more different exponential distributions
  • measurement error
  • model error

(I'm not looking to estimate the parameters of this model. I describe this only to inform the potential distributions of the data, and what the summary descriptive-statistics need to handle.)

As such, the data is:

  • never negative.
  • not symmetric, and likely right-skewed.

I currently have:

  • Five-number summary: minimum, Q1, median, Q3, maximum (normalized and shifted by the median)
  • IQR
  • MAD (median, from the median)

The normalized and shifted five-number summary gives a rough idea of the distribution of the values about the median "center". (If I were to graph these values, I'd graph them un-shifted and un-scaled with the time-axis always starting at zero; the spacing between the values on the graph would give the sense of scale/spread.) Overall, I'm content with these values.

The IQR is nice, since it describes the center 50% of the data. Yet, since this is a skewed distribution, the center 50% of the data will not necessarily describe the peak or the densest part of the data. (Consider the case where one exponential distribution dominates the rest of the model factors; the peak is the leftmost value, and the [min,median] contains the densest part of the distribution.)

The MAD seems to be nicer than the IQR, as it (empirically) always measures a smaller distance than the IQR half-range. Yet, it represents a symmetric range about the median, which also seems to ignore the skewedness aspect.

So, both IQR and MAD measure something, but it seems to ignore the skewness of the distribution. Or, IQR and MAD seem like fine measures for symmetric distributions; in those cases, they will approximate each other, which for the Gaussian (error) Distribution will be centered on the peak value (mu), and contain the densest 50% of the distribution.

Given my critique above of the IQR and MAD, I'm considering identifying the narrowest range that contains 50% of the samples. This calculation can be done in linear-time, and has an easy to explain meaning. But, I'm unaware of anyone else using such a measurement, or what it should be called.

So, given my data model with non-negative, non-symmetric (likely right-skewed) values, my requirement for a linear data calculation, and my desire for a meaningful, descriptive statistic for scale, what alternatives should I consider?

Addendum:

Since it may come up in comments/answers, I am using multiple measures of location, since each has a distinct purpose:

  • minimum: mostly robust; useful as it's the most "optimistic" answer in the absence of system variability.
  • median, with 9X% confidence-intervals: very robust; useful to say 50% of samples are within a certain range of the minimum, in the presence of system noise.
  • mean: somewhat fragile; useful for estimating aggregate behaviour over a sequence of repeated operations.
  • maximum: very fragile; useful for doing worst-case analysis.
  • (I'm not using mode, this seems very fragile to system noise.)
$\endgroup$
  • $\begingroup$ How many data do you have? Is it inexpensive to run the system and get multiple data? How many data could you have? Could you post a small example dataset for people to work with? $\endgroup$ – gung Mar 14 '17 at 15:36
  • $\begingroup$ @gung Thanks for pointing this out; I tried to correct that paragraph. In my program, I usually print 'Q3-med', and 'med-Q1', which I compare to the MAD value, so I sometimes forget MAD is a half-range or radius, and IQR is a full-range or a "diameter". $\endgroup$ – Charles L Wilcox Mar 14 '17 at 15:39
  • $\begingroup$ What will "characterizing" the scale do for you / what do you want to do w/ this measure? Eg, are there constraints like you're reporting this to someone else w/ a specific background & expectations or limited statistical familiarity? Could you use multiple numbers (eg, a 5-point summary)? $\endgroup$ – gung Mar 14 '17 at 15:40
  • 1
    $\begingroup$ @gung It varies, from thousands to millions. $\endgroup$ – Charles L Wilcox Mar 14 '17 at 16:49
  • 1
    $\begingroup$ Strongly related: Mean±SD or Median±MAD to summarise a highly skewed variable? $\endgroup$ – gung Mar 15 '17 at 18:33
0
$\begingroup$

The Highest Density Internal (HDI), a concept used in Bayesian Statistics, is the narrowest internal that contains XX% samples from a distribution. Using a 50% boundary should be maximally robust, having a 50% breakdown point. The only assumption behind it is that the distribution is unimodal, which may not be true for the general case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.