I am working with a dataset representing a material's 'Range of Coverage', or, a calculated amount of time it is expected to stay in stock. This calculation is based on a material's usage during a period of time. So far, my values have been less than 150; however, I am running into an issue during periods with no material usage as they shoot the calculation up to 99,999. When I analyze this data in my statistical software, the extreme, essentially meaningless, values of these outliers greatly affect my test results. Can anyone recommend a way to 'dilute' these outliers due to their value but at the same time utilize their presence as a statistical measure? In other words, I do not want to completely remove them because their presence is significant but since the value of each observation is being analyzed, what other options do I have to test this dataset?

Here is an example dataset which includes both cases - a ROC value of '0' as well as a ROC value of '99,999'. Since these values represent the approximate number of inventory days-on-hand, '0' makes sense but '99,999' does not.

Feb-13 - 99,999 Mar-13 - 0 Apr-13 - 34 May-13 - 0 Jun-13 - 99,999 Jul-13 - 44 Aug-13 - 18 Sep-13 - 99,999

Graphical statistics based on Log of dataset

Thank you all! MCC

  • 2
    $\begingroup$ A very simple expedient to consider is to re-express the values as reciprocal times. In many (but certainly not all) cases where your problem is encountered (essentially a division by zero), that is an indication that such a re-expression might be a better way to understand, summarize, and analyze the data. $\endgroup$ – whuber May 29 '14 at 19:04
  • $\begingroup$ In this case, I do not believe using the reciprocal will work since a period with a Range of Coverage of '0' is significant; if I take the reciprocal of my 99,999 values, the results is essentially '0' as well. $\endgroup$ – Gryphoenix May 30 '14 at 12:47
  • $\begingroup$ @whuber I transformed my data by calculating its Log but my 'outliers' are still majorly affecting my distribution. Any ideas? $\endgroup$ – Gryphoenix May 30 '14 at 16:30

I think that, rather than dilute the outliers in the data as you have it, it would be better to make the data not have those outliers. It appears that the reason you have them is that there are certain periods when there is no purchasing of the material (or almost none). You then take a ratio of 1/0 or 1/(some very small number) and get a huge predicted length of time.

Two ideas:

  1. Use a longer period of time.

  2. Use a weighted moving average of several periods of time.

The former would be simpler to implement but might miss some important information; the latter shouldn't be that hard to do.

  • $\begingroup$ Yes, the reason I have those outliers is because there was no usage during that period. I will work on your two suggestions. $\endgroup$ – Gryphoenix May 29 '14 at 18:07
  • $\begingroup$ Using a 3-period weighted moving average does not appear to help me as I will still have some exceptional values which do not accurately reflect the original calculations. $\endgroup$ – Gryphoenix May 29 '14 at 18:14

I believe that for this exercise, my best approach will be to replace the extreme outliers with values 3 standard deviations from the remaining values' sample mean. This will at least alert me that my dataset has some periods which show no usage.

  • $\begingroup$ In most applications this will bias the results and in some applications it can have terrible effects. For instance, it will not always flag the outliers unambiguously: up to one-ninth of the good data can exceed 3 SDs. Your statistical software, if it is any good at all, can represent missing values (and flag suspect values) and it will handle such values automatically in a meaningful way. $\endgroup$ – whuber May 29 '14 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.