12
$\begingroup$

I'm working on statistics for software builds. I have data for each build on pass/fail and elapsed time and we generate ~200 of these/week.

The success rate is easy to aggregate, I can say that 45% passed any given week. But I'd like to aggregate elapsed time as well, and I want to make sure I don't misrepresent the data too badly. Figured I'd better ask the pros :-)

Say I have 10 durations. They represent both pass and fail cases. Some builds fail immediately, which makes duration unusually short. Some hang during testing and eventually time out, causing very long durations. We build different products, so even successful builds vary between 90 seconds and 4 hours.

I might get a set like this:

[50, 7812, 3014, 13400, 21011, 155, 60, 8993, 8378, 9100]

My first approach was to get the median time by sorting the set and picking the mid-value, in this case 7812 (I didn't bother with the arithmetic mean for even-numbered sets.)

Unfortunately, this seems to generate a lot of variation, since I only pick out one given value. So if I were to trend this value it would bounce around between 5000-10000 seconds depending on which build was at the median.

So to smooth this out, I tried another approach -- remove outliers and then calculate a mean over the remaining values. I decided to split it into tertiles and work only on the middle one:

[50, 60, 155, 3014, 7812, 8378, 8993, 9100, 13400, 21011] ->
[50, 60, 155], [3014, 7812, 8378, 8993], [9100, 13400, 21011] ->
[3014, 7812, 8378, 8993]

The reason this seems better to me is two-fold:

  • We don't want any action on the faster builds, they're already fine
  • The longest builds are likely timeout-induced, and will always be there. We have other mechanisms to detect those

So it seems to me that this is the data I'm looking for, but I'm worried that I've achieved smoothness by removing, well, truth.

Is this controversial? Is the method sane?

Thanks!

|cite|improve this question
$\endgroup$
  • $\begingroup$ You still have not said why you want to have your measure of elapsed time. Do you need to estimate the total machine time you are paying for? Do you need to estimate how many trips to the coffee machine you'd be able to make? What is the action that you will likely take if you knew the times? $\endgroup$ – StasK Aug 28 '14 at 19:43
  • $\begingroup$ Also, if you have heterogeneous products that you build, then bouncing around is natural, and you should not try to get rid of it. If you have a measure of complexity of a given project (# of files, # of lines of code, # of routines, etc.), then you may be interested in elapsed time normalized by complexity, which may be more stable. If you want to make it really stable and flat over time, then you won't be able to learn when you need to take your actions. $\endgroup$ – StasK Aug 28 '14 at 19:45
  • $\begingroup$ @StasK: thanks for your comments, this is valuable input. The action is already known: many of these builds take waaaay too long, and we should work on making them faster. I want to use the aggregate measure to see if things are getting better or worse. $\endgroup$ – Kim Gräsman Aug 30 '14 at 13:43
  • $\begingroup$ And yes, ideally I would probably do this by product, but the source data doesn't maintain that information so I cheat based on what I know about the distribution. This might change over time, of course, and I may have to re-think. $\endgroup$ – Kim Gräsman Aug 30 '14 at 13:45
  • $\begingroup$ Well it looks to me that you have three groups in your data: (i) builds that build; (ii) builds that fail (and you say they tend to fail rather quickly), and (iii) builds that hang up (or nearly so) -- and that's probably the main if not the only group you want to take action upon. The aggregate measure of time may not be of great use; but something like the 10th longest build, or the 10th highest percentile of the distribution of build times, might be a better measure of what's at the extreme of the distribution that you care about. That's roughly what EPA uses in their pollution monitoring. $\endgroup$ – StasK Sep 2 '14 at 23:08
8
$\begingroup$

Your approach makes sense to me, taking your goal into account. It's simple, it's straightforward, it gets the job done, and you likely don't want to write a scientific paper about it.

One thing that one should always do in dealing with outliers is to understand them, and you already do a great job about this. So possible ways of improving your approach would be: can you use info on which builds are hanging? You mention that you have "other mechanisms to detect those" - can you detect them and then remove only those from the sample?

Otherwise, if you have more data, you could think about removing not tertiles, but quintiles... but at some point, this will likely not make much of a difference.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ When I say "other" mechanisms, I mean completely different outside of this data set, so I can't make decisions based on it here. We're aggregating data daily, and we only get ~50 builds per day, so I think quintiles might yield too little data, but thanks for the suggestion, I can experiment with it! $\endgroup$ – Kim Gräsman Aug 28 '14 at 7:14
13
$\begingroup$

What you're doing is known as a trimmed mean.

As you have done, it's common to trim the same proportion from each side (the trimming proportion).

You can trim anything between 0% (an ordinary mean) up to (almost) 50% (which gives the median). Your example has 30% trimmed from each end.

See this answer and the relevant Wikipedia article.

[Edit: See Nick Cox's excellent discussion on this topic.]

It's quite a reasonable, somewhat robust location estimator. It's generally considered more suitable for near-symmetric distributions than highly skewed ones, but if it suits your purpose* there's no reason not to use it. How much is best to trim depends on the kinds of distribution you have and the properties you seek.

* It's not completely clear what you want to estimate here.

There are a large number of other robust approaches to summarizing the 'center' of distributions, some of which you might also find useful. (e.g. M-estimators might have some use for you, perhaps)

[If you need a corresponding measure of variability to go with your trimmed mean, a Winsorized standard deviation might be of some use to you (essentially, when calculating the s.d., replace the values you would cut off when trimming with the most extreme values you didn't cut off).]

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I haven't actually checked the typical shape of my distribution, I think it varies week-by-week depending on where people make changes. It might in fact be highly skewed. $\endgroup$ – Kim Gräsman Aug 30 '14 at 13:52
  • $\begingroup$ Thanks for putting a name on my hack-job, btw. It inspires some confidence :-) $\endgroup$ – Kim Gräsman Aug 30 '14 at 13:52
2
$\begingroup$

Yet another method is to compute the median of all pairwise averages or do bootstrapping.

Update:

The median of all pairwise avarages is called the Hodges–Lehmann estimator. This estimator has usually a high efficiency. This encyclopedia entry by Scott L. Hershberger says:

While both the median and Hodges-Lehmann estimator are both preferable to the sample mean for nonsymmetric distributions, the Hodges-Lehmann estimator has larger asymptotic relative efficiency with respect to the mean than the median

Bootstrapping may be less relevant and more computational intensive, but you could take a small random sample of the data with replacement and compute the mean of that subsample, do it many times and compute the median of all the means.

In both cases you do no longer have to pick a value among the values of your data (when you compute the ordinary median), but instead you pick among many averages from subsets of the data.

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ Can you describe advantages and disadvantages of these methods? A one-line answer is an insufficient one. $\endgroup$ – StasK Aug 28 '14 at 19:46
  • 1
    $\begingroup$ I have tried to make an update with more explanation $\endgroup$ – Finn Årup Nielsen Aug 29 '14 at 0:00
  • $\begingroup$ Both pairwise averages and bootstrapping seem to involve a number of recalculations over the whole dataset. The sample is generally pretty small (<500 values), so it's probably not a problem, but it does seem more complicated. Good to know there are more approaches to experiment with if the trimmed mean turns out to be too blunt. Thanks! $\endgroup$ – Kim Gräsman Aug 30 '14 at 14:14
1
$\begingroup$

Seems reasonable what you are doing : just for information I use the following process quite often for a similar purpose: but I'm only ever really interested in the Upper Outliers.

Calculate five number summary: Min, Q1, Median, Q3, Max. Calculate Interquartile Range : Q3-Q1. Set your outlier 'fences' at Q1-IQR*X, and Q3+IQR*X: where a reasonable value of 'X' is 1.5.

Using Excel and your figures the above (using 1.5 for 'X' **) yields one upper outlier :21011

MIN 50
Q1  3014
MEDIAN  8095
Q3  9073.25
MAX 21011
IQR 6059.25
UPPER FENCE 18162.125
LOWER FENCE -6074.875

So the lower fence here isn't useful or realistic for your example in fact: which backs up the point made by the other post regarding the importance of understanding the meaning of your specific data.

(**Found one citation for the '1.5' rule: I'm not saying it is authoritative, but is seems a reasonable starting point to me : http://statistics.about.com/od/Descriptive-Statistics/a/What-Is-The-Interquartile-Range-Rule.htm)

You could also decide (perhaps) just to use the data points that fall within the IQR itself : this seems to yield sensible results (in that the membership to your method is very similar).

using the same data, this would place the following data points in the 'area of interest':

7812
3014
13400
21011
8993
8378
9100

On a Boxplot : these points would all fall within the box-part (rather than the whiskers-part) of the diagram.

At can be seen that this list includes some items not in your original list (the longer running builds); I can't say whether one list is more accurate in any way. (again, comes down to understanding your dataset).

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks, this seems very close in principle to what I'm already doing. In what ways might this method be better than just partitioning the data set? $\endgroup$ – Kim Gräsman Aug 31 '14 at 19:13
  • 1
    $\begingroup$ I can't really say whether either method is better or not: I suggested this based on what I have read in stats books /sites. One thing (I think) the method adds is that allows you to adjust the 'fences' to whatever you feel is suitable (adjusting the 1.5 factor) ; where just taking the middle third might be more susceptible to including a potentially high outlier (which arguably could be better classified outside of the fence).....but I cant' say that for sure. One other things about using Quartiles, is that the 5-number summary is quite well understood and commonly used. $\endgroup$ – monojohnny Sep 1 '14 at 11:18
  • $\begingroup$ Thanks for the added detail! I do this in Python, so I already have a function where I partition the dataset into N parts and then pull out the middle part. It's a one-character change to make it 5 or 7 partitions instead. But your point is well taken, and it's good to see another common way of approaching this problem. Thanks again. $\endgroup$ – Kim Gräsman Sep 1 '14 at 15:11
  • $\begingroup$ sounds fine to me : I will add one more thing to emphasize about using quartiles and fences (rather than having variable partitions) - is that they map straight onto boxplots: en.wikipedia.org/wiki/Box_plot which allow (amongst other things) a standard way of comparing different distributions graphically. $\endgroup$ – monojohnny Sep 3 '14 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.