6
$\begingroup$

I am looking at timeseries data in foreign exchange and bond markets (to test for reversion on extreme moves). Unfortunate "tick" data, namely high frequency data, is prone to many problems, and they obviously can significantly mess with the analysis. I'd like to know which R library can help with the following type of fairly frequent data cleaning problems:

1) one spike:

enter image description here

This is typically created when one market maker prints a wrong quote in one tick, but there would have been no tradability at that price because it lasted for a split second. I'd like to eliminate the spike (but only if there is only one (or maybe 2) prints)

2) bid ask gapping:

enter image description here

In this case the market is fairly illiquid and the data algorithm is jumping between bids and asks (in this case 2bps wide) causing this weird cloud.

Where should I start to clean this stuff, obviously trying to throw out the least amount of real data. I realise that the maxim of "look at the data" applies here, but when you're looking at 1000 series each with 100 days of data, you can see how this will become quickly impractical so I need some automated help. I'll also look at Python language methods if they're available or better.

$\endgroup$
7
$\begingroup$

There's a package for that. Check out RTAQ.

Small plug: there's a quantitative finance stack exchange you may be interested in.

$\endgroup$
  • $\begingroup$ Aha! Thanks Joshua for the RTAQ link, and also, I'm probably going to migrate the question. HOw do I do that without rudimentary cut and paste? $\endgroup$ – Thomas Browne Jun 3 '11 at 19:52
  • $\begingroup$ You're welcome. You're asking the wrong person about migrating questions. That's one area (of many) where I'm clueless. $\endgroup$ – Joshua Ulrich Jun 3 '11 at 19:53
  • $\begingroup$ @Thomas Migration is easy, but once done, it's very hard to migrate back. It looks like this question could find a home at either site. If you're sure you want to move this question to Quant.SE, then just flag it and a moderator will take care of the move. $\endgroup$ – whuber Jun 3 '11 at 20:07
  • $\begingroup$ @Whuber okay let me just keep it here then - getting some decent answers. Nice to know about the Quantfi SE though. $\endgroup$ – Thomas Browne Jun 3 '11 at 23:11
4
$\begingroup$

To detect an anomaly, you need a model which provides an expectation. Intervention Detection yields the answer to the question " What is the probability of observing what I observed before I observed it ? I suggest that you focus on shorter time series and use an automatic modeling algorithm that forms an ARIMA model based upon separating signal and noise. This ARIMA model can then used to identify the "unusual". Time Series Methods can be used to alert users that the underlying activity has significantly changed. The problem is that you can't catch an outlier without a model (at least a mild one) for your data. Else how would you know that a point violated that model? In fact, the process of growing understanding and finding and examining outliers must be iterative. This isn't a new thought. Bacon, writing in Novum Organum about 400 years ago said: "Errors of Nature, Sports and Monsters correct the understanding in regard to ordinary things,and reveal general forms. For whoever knows the ways of Nature will more easily notice her deviations; and, on the other hand, whoever knows her deviations will more accurately understand Nature.

$\endgroup$
  • $\begingroup$ yep sounds interesting..... but how much higher frequency should I go? In the bottom chart for example, there are already 6333 ticks over a 10.5 hour period so we're looking at a datapoint every 6 seconds. Actually I think I see where you're going anyway. Since I need about 1-minute resolution for my problem, If I build a sub-minute ARIMA-based prediction and look for outliers I should be able to get this right. Certainly the principals you mention are very relevant and indeed thought provoking (to this middle-aged student). $\endgroup$ – Thomas Browne Jun 3 '11 at 23:06
  • $\begingroup$ @ThomasB If I can help offline please feel free to get in touch with me. $\endgroup$ – IrishStat Jun 3 '11 at 23:33
1
$\begingroup$

I'm not convinced "cleaning" is required. Modeling high frequency data is addressed in the academic quantfi literature using functional forms such as GARCH and especially HARCH for the prediction of short-term volatility. This literature dates back to the 90s, e.g., this paper by Dacorogna, et al., http://long-memory.com/volatility/Dacorogna-etal1998.pdf and others by the same author. In my opinion, this would be a more preferable approach than modifying the data as it retains the information, e.g., as you note in the bid-ask spread.

$\endgroup$
0
$\begingroup$

Mayby try some "moving deletator" - in window of p observations compute standard deviation and then delete obs for which absolute difference to previous observation is x times bigger then standard deviation in that window. But this method could don't work with densely packed outliers (one after another) which is showed on the second picture.

ps. from what program are that pictures ?

$\endgroup$
  • $\begingroup$ Yes I like the moving deletion using the SD. I'll use it thank Qbik. Both of the graphs are from the Bloomberg terminal. $\endgroup$ – Thomas Browne Jun 8 '11 at 8:54
0
$\begingroup$

One note. A lot of the spikes are trades that are printed as part of a derivatives transaction. Most data sources that you will use have a field for condition code. Eliminate all non-continuous market trades. You will still have some spikes but at least they are real.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.