12
$\begingroup$

I'm writing a script that analyses run times of processes. I am not sure of their distribution but I want to know if a process runs "too long". So far I've been using 3 standard deviations of the last run times (n>30), but I was told that this does not provide anything useful if the data is not normal (which it does not appear to be). I found another outlier test that states:

Find the inter quartile range, which is IQR = Q3 - Q1, where Q3 is the third quartile and Q1 is the first quartile. Then find these two numbers:

a) Q1 - 1.5*IQR b) Q3 + 1.5*IQR

The point is an outlier if < a or > b

My data tends to be things like 2sec, 3sec, 2sec, 5sec, 300sec, 4sec, .... where 300sec is obviously an outlier.

Which method is better? The IQR method or the std deviation method?

Improve this question
$\endgroup$
5
  • 4
    $\begingroup$ You may want to check out @user603's answer here: is there a boxplot variant for poisson distributed data for info on how to adjust this rule for skewed data. $\endgroup$
    – gung - Reinstate Monica
    May 28, 2013 at 14:40
  • 3
    $\begingroup$ This "IQR" method was never intended to be applied blindly. It is part of a process of exploratory data analysis (as described by Nick Cox in his answer) during which you would first find a way to re-express the data to make them approximately symmetrically distributed. $\endgroup$
    – whuber ♦
    May 28, 2013 at 14:58
  • 2
    $\begingroup$ Based on your comments to the answers, the correct reply is "neither," because your underlying concern is not about outliers, it's about the process. $\endgroup$
    – whuber ♦
    May 28, 2013 at 16:35
  • $\begingroup$ Related: Detecting outliers using standard deviations is the flip side of this question $\endgroup$
    – user56reinstatemonica8
    Oct 15, 2013 at 11:20
  • $\begingroup$ The numbers are time_taken so they will never be symmetric unless you re-scale them somehow. $\endgroup$
    – JP Bennett
    Jun 4, 2018 at 17:42

2 Answers 2

Reset to default
15
$\begingroup$

There really are entire books on outliers.

The usual specific answer is as that the standard deviation is pulled up by outliers, so any rule based on the SD may perform poorly.

The Tukey rules on quartiles +/- 1.5 IQR you quote came out of handwork with small and moderate-sized datasets in the 1970s, and were designed to indicate values you might want to think about individually. It is not clear that they carry over to much larger datasets, nor that they apply when you expect considerable skewness.

A more general answer is that an outlier rule is good if it always makes the right decisions, but how can you tell?

This is contentious territory, but I'd expect an outlier to stick out on a graph as being very different from others. But it is often (usually?) a tough call to tell the difference between what you expect in a heavy-tailed distribution and what is too wild to regard as anything but an outlier. Sometimes transformation makes an outlier look much more ordinary.

Furthermore, if you use robust methods you might worry a bit less about precisely which values merit being called outliers, but worry rather about outliers in general.

EDIT 20 July 2022 A detail that can easily bite is that the IQR can be 0 without the data being in any sense pathological or bizarre. A simple example is any (0, 1) variable with at least 75% values equal to 0 or at least 75% values equal to 1. Then all values not the majority value could be declared outliers, which doesn't match what a statistically experienced person would usually want. Similar awkwardness can easily arise with ordered categorical or counted variables with a small number of values in practice.

Improve this answer
$\endgroup$
1
$\begingroup$

You say you're not sure of the distribution but processes that are ongoing are easy to collect and assess for distribution. Just save a bunch of times and analyze those. Given the times you posted you could get lots in a few hours.

Your search for a rule for an outlier needn't be so general. It can be specific to your task. You're able to collect lots of data. Collect it, examine it, and then decide when a process is too long. Maybe an IQR based approach will work but you can use your data set, or a parametric fit, to do simulations and see if it works well. The same goes for SD. It might just be that >50s is too long and that's all you need.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I am collecting data on several processes. They may each have different distributions. I just need a simple way to say "running time too great" to alert the technicians to further look into things. It can be general so long as it flags things that should be flagged. If a few false positives appear, so be it. However false positives should be kept to a minimum since if there are too many it defeats the purpose of the script and I should just dump all the results and let the techs have at it. The purpose of the script is to "narrow things down" $\endgroup$
    – chris bedd
    May 28, 2013 at 15:28
  • $\begingroup$ You can assess whether the processes are same or different. If they're really very different some general rule may tend to cause a particular process to trigger the warning more frequently than necessary. This information should really be in your question. $\endgroup$
    – John
    May 28, 2013 at 15:32
  • 4
    $\begingroup$ Characterizing this problem as a search for outliers, chris, does it injustice: you are actually tackling a quality control problem. The principal distinctions are (1) you have an ongoing stream of data rather than a static dataset to analyze and (2) you intend to specify periodic actions to take as a result of each analysis: that is, whether to intervene (and attempt to improve the process) or not (and let the process run as is). Understanding that this is the nature of your problem shows that the huge literature on quality control is relevant, providing a rich assortment of solutions. $\endgroup$
    – whuber ♦
    May 28, 2013 at 16:33
  • $\begingroup$ +1 @whuber . Outliers are not relevant here. Neither the average run time, nor any percentile of it, is related to what is "too long". The way to find out what is "too long" may be a survey of users, or a check with engineers, or just seat of the pants guessing, or something else, but it's not a statistical question. $\endgroup$
    – Peter Flom
    May 28, 2013 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.