7
$\begingroup$

In this case, "generic" being the entire gauntlet of macroeconomic time-series that private and government statistical offices put out.

Some background - I recently started working at a data provider - we collect data releases and repackage them in a presumably more convenient and accessible fashion for our clients, and we have tens of thousands of data series (wouldn't be surprised if we topped a million, actually). As part of our QA process, we run the following outlier detection:

$X_t-X_{t-1} = E_t$
$\sigma^2$ is estimated from the resulting sample of $E_t$, and a z-score is calculated based off $E_t\sim N(0,\sigma^2)$

I think we can do better - the math clearly falls apart for everything that isn't a random walk.

I initially thought of fitting an ARMA(m,n) based on the peak of the autocorrelation/autocovarience functions of the series and checking the residuals. I'm wary of the robustness of this, and a previous question seems to indicate that autocorrelation is not particularly robust.

$\endgroup$
  • 5
    $\begingroup$ what is an outlier? The answer heavily depends on the definition. For example 2008 crisis will be reflected by large drop in quite a few macroeconomic indicators and according to a conventional definitions of outlier (the value which "is far" from "the center") this drop will be an outlier, but definitely not the kind you want to throw out. $\endgroup$ – mpiktas Jun 30 '11 at 6:46
  • $\begingroup$ "Gauntlet" or "gamut"? $\endgroup$ – cardinal Jun 30 '11 at 19:45
  • $\begingroup$ The latter. @mpiktas - Certainly. Of course the entire process isn't automated, but rather flags values that should require attention. What is needed is no intervention in said flagging of values. $\endgroup$ – Affine Jul 1 '11 at 3:27
4
$\begingroup$

You are quite right that the ARIMA Model you are using (first differences) may not be appropriate to detect outliers. Outliers can be Pulses, Level Shifts, Seasonal Pulses or Local Time Trends. You might want to google "INTERVENTION DETECTION IN TIME SERIES" or google "AUTOMATIC INTERVENTION DETECTION" to get some reading matter on INTERVENTION DETECTION. Note that this is not the same as INTERVENTION MODELLING which often assumes the nature of the outlier and does not empirically identify same. Following mpkitas's remarks one would include the empirically identified outliers as dummy predictor series in order to accommodate their impact. A lot of work has been done in identifying oultliers using a null filter and then identifying the appropriate ARIMA Model. Some commercial packages assume that you identify the arima model first ( possibly flawed by the outliers ) and then identify the outliers. More general procedures examine both strategies. Your current procedure follows the "use up front filter first" approach but is also flawed by the assumption of the upfront filter.

Some more reflections: to detect an anomaly, you need a model which provides an expectation. Intervention Detection yields the answer to the question " What is the probability of observing what I observed before I observed it ? AN ARIMA model can then used to identify the "unusual" Time Series observations. The problem is that you can't catch an outlier without a model (at least a mild one) for your data. Else how would you know that a point violated that model? In fact, the process of growing understanding and finding and examining outliers must be iterative. This isn't a new thought. Bacon, writing in Novum Organum about 400 years ago said: "Errors of Nature, Sports and Monsters correct the understanding in regard to ordinary things,and reveal general forms. For whoever knows the ways of Nature will more easily notice her deviations; and, on the other hand, whoever knows her deviations will more accurately understand Nature, The Model you are imposing on all your series i clearly am inadequate way to go.

$\endgroup$
  • $\begingroup$ (+1) I was pretty sure that you will answer your favorite topic :) @IrishStat, may be you have also your preferred reviews (just for those who has no time to dig in search engines). $\endgroup$ – Dmitrij Celov Jun 30 '11 at 11:22
  • $\begingroup$ @Dm: Since you asked 1)An overview stpete.usf.edu/gkearns/Articles_Fraud/Fraud%20Magazine1.pdf ; 2)Details on how to program Intervention Detection autobox.com/pdfs/forestdisturbance.pdf $\endgroup$ – IrishStat Jun 30 '11 at 11:54
  • $\begingroup$ @Dm: For Transparency considerations I should advise you that I am one of the developers of AUTOBOX which is cited in both of the above references. For other commercial sites significantly less powerful in my opinion as they don't detect time trends or seasonal pulsesyou might look at support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/… . $\endgroup$ – IrishStat Jun 30 '11 at 13:01
  • $\begingroup$ @Irish - Will certainly take a look, especially at the citations. I'm actually interested in the actual process (and the theory/math behind it) as I will shortly be pursuing a MS. In addition, give my firm's inclination, we'd probably end up rolling our own, if I can convince them of an improved method. In addition, I kind of wonder what exactly the implications of our current model. For example, if the series is an AR(1), I think the current strategy would run more false positives? $X_t - X_{t-1} \sim N(0, something>\sigma^2)$. Would there be any case where the variance end up lower? $\endgroup$ – Affine Jul 1 '11 at 3:39
  • $\begingroup$ @asdfion: The number of false positives (or false negatives!) depends on the autocorrelative structure AND the appropriateness of the filter being used.Now to precisely answer your question if the series is positively autocorrelated say ar(1) of .5 then your procedure will over estimate the variance of the noise and thus will under estimate (mask) the number of outliers. Good luck on rolling your own!. I have been at it for nearly 50 years. if the series is negatively correlated ar(1)of -.5 my gut reaction is a first differnce filter will also yield an increase in false negatives $\endgroup$ – IrishStat Jul 1 '11 at 10:25
0
$\begingroup$

Winsorization replaces extreme data values with less extreme values. http://www.r-bloggers.com/winsorization/

$\endgroup$
  • 1
    $\begingroup$ If the series looks like 1,9,1,9,1,9,1,9,5,9,1,9,1,9 ,winsoration win't help $\endgroup$ – IrishStat Jul 1 '11 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.