Finding abnormally high/low values

Question

I am looking for technique to identify abnormally low/high values in a set of data. I am using the technique to compare the production of machines. The number of machines in the set can be quite small (5) and range up to 200.

I want to identify problematic machines so as to flag them for inspection, or outstanding machines to use them as reference for maximum potential.

I stumbled across IQR analysis using percentiles to identify outliers. This is nice but not quite what I was looking for. The problem with IQR analysis is that because my dataset is so small, it always finds outliers. If the production of my machines is more or less the same there will always be machines in the low and high quarters of the sample.

I am leaning towards a standard deviation approach to determine if there are any exceptionally high or low alues within the set. But I don't know how many standard deviations to count and I am looking for a generic method which can apply accross different sites (ie, maybe the number of standard deviations to count is different for different sites ???)

My data set consists of a performance-ratio between 0 and 1 :

• Typical values range from 75% to 90%
• It will vary throughout the year and from machine to machine as it is dependant on a range of factors.

Sample data:

Here is some real data and the results of my current analysis:

Machine 1:  74.1%    12th percentile
Machine 2:  50.2%    6th percentile      Machine underperforming - should be flagged
Machine 3:  76.5%    32th percentile
Machine 4:  78.9%    78th percentile
Machine 5:  78.9%    78th percentile
Machine 6:  74.1%    12th percentile
Machine 7:  76.5%    32th percentile
Machine 8:  0%       0th percentile      Machine broken down - should be flagged
Machine 9:  78.9%    78th percentile
Machine 10: 76.5%    32th percentile
Machine 11: 72.2%    10th percentile
Machine 12: 76.5%    34th percentile
Machine 13: 78.7%    75th percentile
Machine 14: 78.7%    75th percentile
Machine 15: 76.5%    34th percentile
Machine 16: 76.5%    34th percentile
Machine 17: 76.5%    34th percentile
Machine 18: 78.7%    75th percentile
Machine 19: 76.5%    34th percentile
Machine 20: 78.7%    75th percentile
Machine 21: 76.5%    34th percentile
Machine 22: 76.5%    34th percentile
Machine 23: 76.5%    32th percentile
Machine 24: 81.3%    100th percentile    Best performing machine, but by no means an outlier
Machine 25: 76.5%    32th percentile
Machine 26: 75.9%    22th percentile
Machine 27: 75.2%    19th percentile
Machine 28: 78.3%    73th percentile
Machine 29: 78.3%    73th percentile
Machine 30: 77.7%    66th percentile
Machine 31: 78.3%    73th percentile

The method I am using:

Once plotted on that number line, the smallest data point and the biggest data point in the set of data create the boundaries of an interval of space on the number line that contains all data points in the set. The interquartile range (IQR) is the length of the middle 50% of that interval of space.

I am using a method which groups the data into 3 groups :

1. The 50% of this range, ranging from 25% to 75% IQR.
2. The lower quartile <25th percentile
3. The upper quartile >75th percentile

This method is insufficient - Machine 27 is flagged as it is in the 19th percentile, while its performance ratio is comperable to other machines in the middle of the range which are not flagged.

However, I am not doing the calculation. I am using an external code library: Apache Commons Math to find the cumulative probability of an emperical distribution. This is where the percentile values are coming from in the above data.

Bonus points:

I would also like to show a simple indicator to determine the trend of the performance ratio. If I was to take for example the daily performance-ratio of a machine over 30 days. How could I determine if it was improving, steady or descending.

Answer 1

It's not clear to me that you should be doing this exercise for each new set of data as though it is fresh. Although it varies from year to year and machine to machine, aren't you interested in outlier deviation from overall performance, not just from the other performers in the batch? eg what if everything in the batch is bad, shouldn't they all be flagged? If this thinking is correct, then you should treat everything as a single (growing) dataset, rather than a fresh dataset each time, which reduces some of your difficulties.

Also, I don't think you are implementing IQR outlier detection as it usually is. If I understand your edits after @whuber's comment correctly, you are classifying everything in the bottom quarter and the top quarter as suspect. As you point out, this obviously causes problems of too-generous identification, and it's not usually how IQR comparisons are used for outlier detection (see below).

Anyway, putting those issues aside, here are some options. This content is taken from chapter 2 of Wilcox's Modern Statistics for the Social and Behavioral Sciences.

1. difference from mean / standard deviation - not recommended

As you hint, this is sometimes used to find outliers. A usual cut-off point is two standard deviations. In your case, this isn't sensitive enough, only machine 8 is identified:

> which(abs(x - mean(x) ) / sd(x) > 2)
[1] 8

The issue is that with the mean and (in particular) the standard deviation being calculated from the data, they are vulnerable to outliers themselves. In this case, the machine with 0 performance results in a large estimated standard deviation, which means many points are within 2 deviations of the mean. This is unlikely to be sensitive enough for you.

2. Boxplot rule - ok

x is an outlier if

$X < q_1 - 1.5(IQR)$ or

$X > q_2 + 1.5(IQR)$

Where $q_1$ and $q_2$ are the so-called lower and upper "ideal fourths" - basically estimates of the 25th and 75 percentiles and $IQR = q_2-q_1$. The formulae for $q_1$ and $q_2$ are surprisingly complex, to deal with the situation where there are relatively few observations, so I won't include them here.

Basically, this says that if you are 1.5 IQRs above or below the bottom or top quartile, you are a suspect/outlier. R's default method (method 7 of 9...) for estimating quartiles says your lower quartile cutoff is 76.5 and the upper is 78.5, so your IQR is 2. You have three suspect points by this method:

> which(x < (quantile(x)[2] - IQR(x) * 1.5))
[1]  2  8 11

3. MAD - Median rule - ok

I quite like this one. This rule proposes x to be an outlier if

$ \frac{|x-median(x)|}{MADN} > 2.24$

where MADN is the adjusted median absolute deviation of x from its median:

$ MADN = \frac{median(|x-median(x)|)}{0.6745}$

(Dividing by 0.6745 is for convenience, so the expected value of $MADN(x)$ becomes the standard deviation of x, for large sample size if x is normally distributed)

In other words, you are an outlier if you are too far away from the median (which is a very robust measure of the middle of the data), where "too far away" is itself defined in a robust way. With your data, this method returns machines 2 and 8 as the outliers:

> which(abs(x - median(x)) / mad(x) > 2.24)
[1] 2 8

BTW here is my definition of the machines' measurements as x, for reproducibility and checking (I could easily have typos, of course).

x <- c(741, 502, 765, 789, 789, 741, 765, 0, 789, 765, 722, 765, 787, 787, 765, 765, 765, 
      787, 765, 787, 765, 765, 765, 813, 765, 759, 752, 783, 783, 777, 783) / 10