Outlier Detection in Time-Series: How to reduce false positives?

I'm trying to automate outlier detection in time-series and I used a modification of the solution proposed by Rob Hyndman here.

Say, I measure daily visits to a website from various countries. For some countries where the daily visits are a few hundrends or thousands, my method seems to be working reasonably.

However, in cases where a country leads to only 1 or 2 visits per day, the limits of the algorithm are very narrow (e.g. 1 ± 0.001) and therefore the 2 visits are considered an outlier. How could I automatically detect such cases and how could I treat them to identify outliers? I wouldn't like to set a manual threshold of, say, 100 visits per day.

Thank you!

• A natural, simple way that might solve your problem--which is due (at least in part) to a highly variable variance--is to apply a variance-stabilizing transformation, such as an Anscombe or Freeman-Tukey transformation, to the data before looking for outliers.
– whuber
Oct 17, 2014 at 16:13

Don't expect much for small, discrete counts. Going from 1 to 2 visits is a 100% increase, and going from 0 to 1 visits is an infinite increase. At low levels you may be dealing with zero-inflated models, and it can be very noisy down there as well.

In my experience, count data with a mixture of large and small counts like this results in two problems with your small counts: 1) they are too coarse to do much with, 2) they are generated by different processes. (Think small, rural post office versus big city post office). So you need to at least split your modeling in two: do what you're successfully doing for the larger counts, and do something different -- coarser and more approximate -- with small counts. But don't expect much of the small counts.

The good news is that the big counts, by definition, include more of your transactions, so your better model covers more of the data, even though it may not cover most of your sites.

(I say "modeling" to be general, but of course outlier detection is assuming a particular model and finding points that are highly unlikely with that model's assumptions.)

Each value from your time series is a sample from a probability distribution. You need to first find what the probability distribution is and then define what the word rare means within that distribution.

So calculate the empirical cdf, and calculate the 95% confidence interval. Whenever something outside of that region has occurred, then by definition you know that it must be a rare event.

You are having that problem because your data is far from a normal distribution. If the distribution is highly asymmetrical, with bumps, humps or too long/short tails you will encounter problems. A good idea is to apply a transformation like Box Cox or Yeo-Johnson before using your method. In your example if you use F(x) = log(1+x) you avoid the different magnitude problem and you can convert back using: exp(z) -1

There are several procedures you could use to find automatically a good lambda for the Box-Cox transformation. I personally use the median of all the methods of the boxcoxnc function from AID package in R. If your data is not strictly positive you will need to add 1 or other positive number before using it.

It is one thing to detect an Outlier at a particular level of confidence and yet another to place a second specification that would further restrict the acceptance of the outlier. I was once asked the following question " Can AUTOBOX detect a mean shift of xx units at a pre-specified level of confidence". Essentially what was required was a dual test. AUTOBOX is a piece of software that I have helped develop which you might find cost-effective as no free software has implemented this dual test.

Thanks Nick: I was using a level shift as a particular example of an "outlier" or in general the empirically identified deterministic impact. Other forms of "outliers" are Pulses, Seasonal Pulses and Local Time Trends AND particular combinations such as a transient change to a new level. The main point was that there may be two hypothesis that are in play reflecting statistical significance and real-world significance. The customer that had originally brought this problem to my attention was interested in both.

• An outlier need not imply a mean shift.... In fact, stepped or ramped change between regimes with different means need not entail outliers at all. You know this very well, but what I think it implies is that it would be most helpful to others to flesh out your answer. Oct 17, 2014 at 14:38
• Thank you both. I'm interested in real world significance. After identifying an outlier, I weight it's significance with, say, the proportion of visits compared to total visits to get it's final significance. Although that proportion is really small in the cases of 2 visits, since the expected values are 1 ± 0.001, the 'distance' of the actual point from the expected limits is very high (e.g. 2/0.002 where 0.002 is the 'iqr'). So, te issue gets high significance in the end. Any ideas? Oct 17, 2014 at 15:22
• I'd flag that the extended use of "outlier" here is much wider even than the generous sense of anything extreme that is common in many literatures. Seasoned readers will know that @IrishStat sticks to his forte, time series analysis. Oct 17, 2014 at 16:36