Good form to remove outliers?

Question

I'm working on statistics for software builds. I have data for each build on pass/fail and elapsed time and we generate ~200 of these/week.

The success rate is easy to aggregate, I can say that 45% passed any given week. But I'd like to aggregate elapsed time as well, and I want to make sure I don't misrepresent the data too badly. Figured I'd better ask the pros :-)

Say I have 10 durations. They represent both pass and fail cases. Some builds fail immediately, which makes duration unusually short. Some hang during testing and eventually time out, causing very long durations. We build different products, so even successful builds vary between 90 seconds and 4 hours.

I might get a set like this:

[50, 7812, 3014, 13400, 21011, 155, 60, 8993, 8378, 9100]

My first approach was to get the median time by sorting the set and picking the mid-value, in this case 7812 (I didn't bother with the arithmetic mean for even-numbered sets.)

Unfortunately, this seems to generate a lot of variation, since I only pick out one given value. So if I were to trend this value it would bounce around between 5000-10000 seconds depending on which build was at the median.

So to smooth this out, I tried another approach -- remove outliers and then calculate a mean over the remaining values. I decided to split it into tertiles and work only on the middle one:

[50, 60, 155, 3014, 7812, 8378, 8993, 9100, 13400, 21011] ->
[50, 60, 155], [3014, 7812, 8378, 8993], [9100, 13400, 21011] ->
[3014, 7812, 8378, 8993]

The reason this seems better to me is two-fold:

• We don't want any action on the faster builds, they're already fine
• The longest builds are likely timeout-induced, and will always be there. We have other mechanisms to detect those

So it seems to me that this is the data I'm looking for, but I'm worried that I've achieved smoothness by removing, well, truth.

Is this controversial? Is the method sane?

Thanks!

Answer 1

Your approach makes sense to me, taking your goal into account. It's simple, it's straightforward, it gets the job done, and you likely don't want to write a scientific paper about it.

One thing that one should always do in dealing with outliers is to understand them, and you already do a great job about this. So possible ways of improving your approach would be: can you use info on which builds are hanging? You mention that you have "other mechanisms to detect those" - can you detect them and then remove only those from the sample?

Otherwise, if you have more data, you could think about removing not tertiles, but quintiles... but at some point, this will likely not make much of a difference.

Answer 2

What you're doing is known as a trimmed mean.

As you have done, it's common to trim the same proportion from each side (the trimming proportion).

You can trim anything between 0% (an ordinary mean) up to (almost) 50% (which gives the median). Your example has 30% trimmed from each end.

See this answer and the relevant Wikipedia article.

[Edit: See Nick Cox's excellent discussion on this topic.]

It's quite a reasonable, somewhat robust location estimator. It's generally considered more suitable for near-symmetric distributions than highly skewed ones, but if it suits your purpose* there's no reason not to use it. How much is best to trim depends on the kinds of distribution you have and the properties you seek.

* It's not completely clear what you want to estimate here.

There are a large number of other robust approaches to summarizing the 'center' of distributions, some of which you might also find useful. (e.g. M-estimators might have some use for you, perhaps)

[If you need a corresponding measure of variability to go with your trimmed mean, a Winsorized standard deviation might be of some use to you (essentially, when calculating the s.d., replace the values you would cut off when trimming with the most extreme values you didn't cut off).]

Answer 3

Yet another method is to compute the median of all pairwise averages or do bootstrapping.

Update:

The median of all pairwise avarages is called the Hodges–Lehmann estimator. This estimator has usually a high efficiency. This encyclopedia entry by Scott L. Hershberger says:

While both the median and Hodges-Lehmann estimator are both preferable to the sample mean for nonsymmetric distributions, the Hodges-Lehmann estimator has larger asymptotic relative efficiency with respect to the mean than the median

Bootstrapping may be less relevant and more computational intensive, but you could take a small random sample of the data with replacement and compute the mean of that subsample, do it many times and compute the median of all the means.

In both cases you do no longer have to pick a value among the values of your data (when you compute the ordinary median), but instead you pick among many averages from subsets of the data.

Answer 4

Seems reasonable what you are doing : just for information I use the following process quite often for a similar purpose: but I'm only ever really interested in the Upper Outliers.

Calculate five number summary: Min, Q1, Median, Q3, Max. Calculate Interquartile Range : Q3-Q1. Set your outlier 'fences' at Q1-IQR*X, and Q3+IQR*X: where a reasonable value of 'X' is 1.5.

Using Excel and your figures the above (using 1.5 for 'X' **) yields one upper outlier :21011

MIN 50
Q1  3014
MEDIAN  8095
Q3  9073.25
MAX 21011
IQR 6059.25
UPPER FENCE 18162.125
LOWER FENCE -6074.875

So the lower fence here isn't useful or realistic for your example in fact: which backs up the point made by the other post regarding the importance of understanding the meaning of your specific data.

(**Found one citation for the '1.5' rule: I'm not saying it is authoritative, but is seems a reasonable starting point to me : http://statistics.about.com/od/Descriptive-Statistics/a/What-Is-The-Interquartile-Range-Rule.htm)

You could also decide (perhaps) just to use the data points that fall within the IQR itself : this seems to yield sensible results (in that the membership to your method is very similar).

using the same data, this would place the following data points in the 'area of interest':

7812
3014
13400
21011
8993
8378
9100

On a Boxplot : these points would all fall within the box-part (rather than the whiskers-part) of the diagram.

At can be seen that this list includes some items not in your original list (the longer running builds); I can't say whether one list is more accurate in any way. (again, comes down to understanding your dataset).