Outlier Detection in Time-Series: How to reduce false positives?

Question

I'm trying to automate outlier detection in time-series and I used a modification of the solution proposed by Rob Hyndman here.

Say, I measure daily visits to a website from various countries. For some countries where the daily visits are a few hundrends or thousands, my method seems to be working reasonably.

However, in cases where a country leads to only 1 or 2 visits per day, the limits of the algorithm are very narrow (e.g. 1 ± 0.001) and therefore the 2 visits are considered an outlier. How could I automatically detect such cases and how could I treat them to identify outliers? I wouldn't like to set a manual threshold of, say, 100 visits per day.

Thank you!

Answer 1

Don't expect much for small, discrete counts. Going from 1 to 2 visits is a 100% increase, and going from 0 to 1 visits is an infinite increase. At low levels you may be dealing with zero-inflated models, and it can be very noisy down there as well.

In my experience, count data with a mixture of large and small counts like this results in two problems with your small counts: 1) they are too coarse to do much with, 2) they are generated by different processes. (Think small, rural post office versus big city post office). So you need to at least split your modeling in two: do what you're successfully doing for the larger counts, and do something different -- coarser and more approximate -- with small counts. But don't expect much of the small counts.

The good news is that the big counts, by definition, include more of your transactions, so your better model covers more of the data, even though it may not cover most of your sites.

(I say "modeling" to be general, but of course outlier detection is assuming a particular model and finding points that are highly unlikely with that model's assumptions.)

Answer 2

Each value from your time series is a sample from a probability distribution. You need to first find what the probability distribution is and then define what the word rare means within that distribution.

So calculate the empirical cdf, and calculate the 95% confidence interval. Whenever something outside of that region has occurred, then by definition you know that it must be a rare event.

Answer 3

You are having that problem because your data is far from a normal distribution. If the distribution is highly asymmetrical, with bumps, humps or too long/short tails you will encounter problems. A good idea is to apply a transformation like Box Cox or Yeo-Johnson before using your method. In your example if you use F(x) = log(1+x) you avoid the different magnitude problem and you can convert back using: exp(z) -1

There are several procedures you could use to find automatically a good lambda for the Box-Cox transformation. I personally use the median of all the methods of the boxcoxnc function from AID package in R. If your data is not strictly positive you will need to add 1 or other positive number before using it.

Answer 4

It is one thing to detect an Outlier at a particular level of confidence and yet another to place a second specification that would further restrict the acceptance of the outlier. I was once asked the following question " Can AUTOBOX detect a mean shift of xx units at a pre-specified level of confidence". Essentially what was required was a dual test. AUTOBOX is a piece of software that I have helped develop which you might find cost-effective as no free software has implemented this dual test.

Thanks Nick: I was using a level shift as a particular example of an "outlier" or in general the empirically identified deterministic impact. Other forms of "outliers" are Pulses, Seasonal Pulses and Local Time Trends AND particular combinations such as a transient change to a new level. The main point was that there may be two hypothesis that are in play reflecting statistical significance and real-world significance. The customer that had originally brought this problem to my attention was interested in both.