I have a time series where I need to detect gross anomalies due to coding errors, not small shifts in the structure of the series. I am interested in the most recent data points, not historical data so I don't need to filter everything. What is the best way to use moving means and averages in combination with spread estimators to find these points?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ Welcome to our community, Georgette! $\endgroup$– whuber ♦Commented Feb 20, 2012 at 14:36
-
$\begingroup$ I recently faced a very similar issue and this is the answer I got from the experts here at CrossValidated. I'd say Whuber's idea is pretty much in line with this (i.e. methods based on finding outlying residuals from robust smooths), so +1 to that (for some reason I can't add a comment directly to his answer). $\endgroup$– BruderCommented Feb 20, 2012 at 16:25
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
4
Simple methods based on finding outlying residuals from robust smooths tend to work well. There are many; I have had good success using Tukey's smoothers (which include windowed medians), but even Lowess (which is computationally intensive but now widely available and very flexible: it can be adjusted for covariates) usually works just fine, providing you aren't working with censored data.
As with all things outlying, there is no omnibus approach, because the nature of the data matters and, also, one person's "gross anomaly" is another person's normal situation;so it helps to have a battery of such procedures available. (Avoid methods based on moving means and standard deviations: their lack of robustness makes them unsuitable for general outlier detection and they will quickly break down when confronted with multiple outlying values within a window.)
Lowess (aka Loess) is available in R (and many commercial packages, such as Systat or Stata). Tukey's smoothers are not widely available, in part because many people deprecate them, but I suspect that may have been due to bad implementations. An Excel macro (which fully implements this class of smoothers, not just a small subset of it as in the Stata implementation) is available from my web site at http://www.quantdec.com/Excel/eda.htm. Tukey describes his smoothers and illustrates their use throughout his EDA textbook (Addison-Wesley, 1977).
-
$\begingroup$ These answers are not answering my question. If I smooth the data with Lowess or a Median Smoother, do I then use a rule like 3 MAD after the smoothing to find out if the last few points are outliers? Also don't either of these smoothers break down at the ends where there is no neighborhood to the right? Or do I just use the MAD of the last N points as an alternative to the standard deviation? This would be simple to implement. $\endgroup$ Commented Feb 21, 2012 at 15:32
-
1$\begingroup$ Typically identifying whether an observation is an outlier and deserves further attention is based in the particular context/objectives and resource constraints. It is possible a particular cut-off (for any particular metric) will flag too many false positive cases (or just too many cases in general to be useful), which would suggest either increasing the cut-off value (or trying a different metric). $\endgroup$– Andy WCommented Feb 21, 2012 at 15:44
-
$\begingroup$ @Georgette Please register your account -- this way you won't lose control of your questions. $\endgroup$– user88Commented Feb 21, 2012 at 16:04
-
$\begingroup$ Tukey pays special attention to the behavior of his smoothers at the endpoints. Although the methods appear ad hoc, they at least address the issue of edge effects in smoothing. $\endgroup$– whuber ♦Commented Feb 21, 2012 at 19:30
$\begingroup$
$\endgroup$
3
In the past I have used Median Absolute Deviation to test for outliers in a static series of data. This is a simple measure to implement and is related to standard deviation for normally distributed data points. It is very robust against outliers. I have used it for very large static data sets rather than with a time series but it should still be an efficient measure to detect outliers.
The median absolute deviation is defined as the median of deviations from the median
$$\text{MAD} = \text{median}_i\left(|X_i - \text{median}_j(X_j)|\right)$$
and you can find out more about it here http://en.wikipedia.org/wiki/Median_absolute_deviation
-
$\begingroup$ +1. This is a special case of looking for outliers in residuals relative to the simplest of Tukey's smoothers, a running median (within a window of $k$ values, with $k$ chosen by the user). $\endgroup$– whuber ♦Commented Feb 20, 2012 at 17:38
-
2$\begingroup$ thanks @whuber. To add, given a dataset where most values follow a normal distribution but where there exists a potentially high proportion of outliers, it will provide an excellent robust measure of the variance of the normal distribution. If the time series data is can be assumed to have a constant mean and variance over time, then the MAD approach can be used to estimate distribution parameters from a sample of the data and in this special case there is no need to have a running median to estimate the mean locally. I guess any implementation will depend on the actual data characteristics. $\endgroup$– martinoCommented Feb 20, 2012 at 22:20
-
$\begingroup$ I will try MAD at various length of runs and cutoffs for outliers. I will compare this with previous approaches. $\endgroup$ Commented Feb 21, 2012 at 17:20
$\begingroup$
$\endgroup$
have a look at the robfilter R package. As usual it contains pointer to supporting documentation.
The package contains efficient online and rolling window based algoritms for computing recursive median, mad, Qn and a host of other robust estimators (this answer complets Matino and Whuber's).
$\begingroup$
$\endgroup$
3
An easy way is to treat as errors whatever deviates more than 3 or 4 standard deviations from the mean of the last N data points.
The choice between 3 or 4 depends on how different an erroneous data point is from a valid data point. The choice of N depends on how much data you have, but a bare minimum is about 6.
answered Feb 20, 2012 at 14:45
-
$\begingroup$ :Carlos unfortunately the standard deviation is effected by "the erroneous data point" thus it is like the fox watching the chickens. Your test assumes a model ( the mean of the last N points which is a particular form of an ARIMA model ) while the "cleansed data" should be suggesting that form via standard time series analytics. $\endgroup$ Commented Feb 22, 2012 at 12:56
-
$\begingroup$ Looks like I wasn't clear enough. Each data point should be judged based on the mean and standard deviations of the N data points that came before the point being "judged". Here's a reference: amazon.com/Understanding-Variation-Key-Managing-Chaos/dp/… $\endgroup$ Commented Feb 22, 2012 at 14:09
-
$\begingroup$ :Carlos Unfortunately you are still assuming the form of the model i.e. an equally weighted average of the last N data points. The optimization of this is to determine BOTH the best "N" and the string of weights to be applied . To assume the weights are equal may appear to be "fair" but it is presumptive. $\endgroup$ Commented Feb 23, 2012 at 12:01
$\begingroup$
$\endgroup$
Although you asked for a robust procedures involving medians, I don't think that you were asking for non-parameteric methods. In my mind the usage of medians suggests a disinclination to develop a parametric-based model. Such a model could include somee form of ARIMA filter in conjunction with Intervention Detection procedures designed to ferret out "anomolous data" To detect outliers you need some sort of a model based upon "regular data". Approaches to do this are well known see Box-Jenkins model selection for some details. One can set a threshold for a "Pulse" and if this threshold is exceeded then one might call this an Outlier/Inlier. These methods can be robustified by accounting for parameter changes/variance changes and of course possible explanatory series.
