Robust outlier detection in financial timeseries I'm looking for some robust techniques to remove outliers and errors (whatever the cause) from financial time-series data (i.e. tickdata). 
Tick-by-tick financial time-series data is very messy. It contains huge (time) gaps when the exchange is closed, and make huge jumps when the exchange opens again. When the exchange is open, all kinds of factors introduce trades at price levels that are wrong (they did not occur) and/or not representative of the market (a spike because of an incorrectly entered bid or ask price for example). This paper by tickdata.com (PDF) does a good job of outlining the problem, but offers few concrete solutions.
Most papers I can find online that mention this problem either ignore it (the tickdata is assumed filtered) or include the filtering as part of some huge trading model which hides any useful filtering steps.
Is anybody aware of more in-depth work in this area?
Update: this questions seems similar on the surface but:


*

*Financial time series is (at least at the tick level) non-periodic.

*The opening effect is a big issue because you can't simply use the last day's data as initialisation even though you'd really like to (because otherwise you have nothing). External events might cause the new day's opening to differ dramatically both in absolute level, and in volatility from the previous day.

*Wildly irregular frequency of incoming data. Near open and close of the day the amount of datapoints/second can be 10 times higher than the average during the day. The other question deals with regularly sampled data.

*The "outliers" in financial data exhibit some specific patterns that could be detected with specific techniques not applicable in other domains and I'm -in part- looking for those specific techniques.

*In more extreme cases (e.g. the flash crash) the outliers might amount to more than 75% of the data over longer intervals (> 10 minutes). In addition, the (high) frequency of incoming data contains some information about the outlier aspect of the situation.

 A: I'll add some paper references when I'm back at a computer, but here are some simple suggestions:
Definitely start by working with returns. This is critical to deal with the irregular spacing where you can naturally get big price gaps (especially around weekends). Then you can apply a simple filter to remove returns well outside the norm (eg. vs a high number of standard deviations). The returns will adjust to the new absolute level so large real changes will result in the loss of only one tick. I suggest using a two-pass filter with returns taken from 1 step and n steps to deal with clusters of outliers.
Edit 1: Regarding the usage of prices rather than returns: asset prices tend to not be stationary, so IMO that can pose some additional challenges. To account for the irregularity and power law effects, I would advise some kind of adjustment if you want to include them in your filter. You can scale the price changes by the time interval or by volatility.  You can refer to the "realized volatility" literture for some discussion on this.  Also discussed in Dacorogna et. al.   
To account for the changes in volatility, you might try basing your volatility calculation from the same time of the day over the past week (using the seasonality).  
A: I have (with some delay) changed my answer to reflect your concern about the lack of 'adaptability' of the unconditional mad/median.
You can address the problem of time varying volatility with the robust statistics framework. This is done by using a robust estimator of the conditional variance (instead of the robust estimator of the unconditional variance I was suggesting earlier): the M-estimation of the GARCH model. Then you will have a robust, time varying estimate of $(\hat{\mu}_t,\hat{\sigma}_t)$ which are not the same as those 
produced by the usual GARCH fit. In particular, they are not driven by a few far away outliers. Because these estimate are not driven by them, you can use them to reliably flag the outliers using the historical distribution of the 
$$\frac{x_t-\hat{\mu}_t}{\hat{\sigma}_t}$$
You can find more information (and a link to an R package) in this paper:

Boudt, K. and Croux, C. (2010). Robust M-Estimation of Multivariate
  GARCH Models.

A: The problem is definitely hard. 
Mechanical rules like the +/- N1 times standard deviations, or +/ N2 times MAD,  or +/- N3 IQR or ... will fail because there are always some series that are different as for example:


*

*fixings like interbank rate may be constant for some time and then jump all of a sudden

*similarly for e.g. certain foreign exchanges coming off a peg

*certain instrument are implicitly spreads; these may be near zero for periods and all of a sudden jump manifold


Been there, done that, ... in a previous job.  You could try to bracket each series using arbitrage relations ships (e.g. assuming USD/EUR and EUR/JPY are presumed good, you can work out bands around what USD/JPY should be; likewise for derivatives off an underlying etc pp.
Commercial data vendors expand some effort on this, and those of use who are clients of theirs know ... it still does not exclude errors.
