Use of robust spread measures such median average deviation and median filters for time series

Question

I have a time series where I need to detect gross anomalies due to coding errors, not small shifts in the structure of the series. I am interested in the most recent data points, not historical data so I don't need to filter everything. What is the best way to use moving means and averages in combination with spread estimators to find these points?

Answer 1

Simple methods based on finding outlying residuals from robust smooths tend to work well. There are many; I have had good success using Tukey's smoothers (which include windowed medians), but even Lowess (which is computationally intensive but now widely available and very flexible: it can be adjusted for covariates) usually works just fine, providing you aren't working with censored data.

As with all things outlying, there is no omnibus approach, because the nature of the data matters and, also, one person's "gross anomaly" is another person's normal situation;so it helps to have a battery of such procedures available. (Avoid methods based on moving means and standard deviations: their lack of robustness makes them unsuitable for general outlier detection and they will quickly break down when confronted with multiple outlying values within a window.)

Lowess (aka Loess) is available in R (and many commercial packages, such as Systat or Stata). Tukey's smoothers are not widely available, in part because many people deprecate them, but I suspect that may have been due to bad implementations. An Excel macro (which fully implements this class of smoothers, not just a small subset of it as in the Stata implementation) is available from my web site at http://www.quantdec.com/Excel/eda.htm. Tukey describes his smoothers and illustrates their use throughout his EDA textbook (Addison-Wesley, 1977).

Answer 2

In the past I have used Median Absolute Deviation to test for outliers in a static series of data. This is a simple measure to implement and is related to standard deviation for normally distributed data points. It is very robust against outliers. I have used it for very large static data sets rather than with a time series but it should still be an efficient measure to detect outliers.

The median absolute deviation is defined as the median of deviations from the median

$$\text{MAD} = \text{median}_i\left(|X_i - \text{median}_j(X_j)|\right)$$

and you can find out more about it here http://en.wikipedia.org/wiki/Median_absolute_deviation

Answer 3

have a look at the robfilter R package. As usual it contains pointer to supporting documentation.

The package contains efficient online and rolling window based algoritms for computing recursive median, mad, Qn and a host of other robust estimators (this answer complets Matino and Whuber's).

Answer 4

An easy way is to treat as errors whatever deviates more than 3 or 4 standard deviations from the mean of the last N data points.

The choice between 3 or 4 depends on how different an erroneous data point is from a valid data point. The choice of N depends on how much data you have, but a bare minimum is about 6.

Answer 5

Although you asked for a robust procedures involving medians, I don't think that you were asking for non-parameteric methods. In my mind the usage of medians suggests a disinclination to develop a parametric-based model. Such a model could include somee form of ARIMA filter in conjunction with Intervention Detection procedures designed to ferret out "anomolous data" To detect outliers you need some sort of a model based upon "regular data". Approaches to do this are well known see Box-Jenkins model selection for some details. One can set a threshold for a "Pulse" and if this threshold is exceeded then one might call this an Outlier/Inlier. These methods can be robustified by accounting for parameter changes/variance changes and of course possible explanatory series.