# Filtering techniques and noise

Suppose we have some house price data for 30 years (1970-1999). This is yearly data (30 data points). Suppose some major event $X$ happened on 1980. I want to see whether this event affected prices later on. Is using median filters a good methodology for detecting outliers in later years?

-
@user603: Let $y_1 = \{1,2,3,4 \}$ and $y_2 = \{1,2,3,4,10,68 \}$. The median filter for the first would be $\{1,2,3,4 \}$ and the median filter for the second would be $\{1,2,3,4,10,68 \}$. –  Damien Jul 24 '12 at 17:29
Meant subsitute i.e. substitute some genuine observations by some outliers and see the result. Also, the #of outliers< T/2-1 (T is the length of your series). –  user603 Jul 24 '12 at 17:56
Damien, I don't think you're computing the filter correctly: you need to be more careful at the ends of the series. A median-of-three filter would give $(\*,2,3,\*)$ and $(\*,2,3,4,10,\*)$ in those cases, with "$\*$" representing a value that shouldn't be trusted. Notice that medians (of 3) give you no defense against pairs of adjacent outliers, either, but wider-window medians will be more stable. –  whuber Jul 24 '12 at 18:04
@Damien Sounds like you could do an intervention analysis. One option would be to use a standard regression model with a binary indicator variable as an independent variable (switched "on" for the period in which the event occurred) and ARIMA errors. Very similar to the ARIMAX / transfer function models mentioned by others, but easier to interpret in terms of effects of the input onto the output. Rob Hyndman has a blogpost on this here. –  Graeme Walsh Oct 19 '13 at 0:37

I like the idea of looking for interventions that form level changes in a process. This can be done using ARIMA models and search for the changepoints. Our friend IrishStat does this with his autobox software as he so nicely demonstated yesterday.

I am sure he will be happy to give you more details and perhaps even do a data analysis for you.

I wouldn't call the level shifts outliers. There are various types of outliers in time series and numerous ways to try to detect them. We also discussed this either yesterday or today. Fox identified two type of outliers in his paper from back in the 70s.

-
There is also a changepoint package in R. –  Damien Jul 24 '12 at 19:09
@Damien There is a changepoint package in R. It is important for you to know all the caveats i.e.all the assumptions underlying the tests that are conducted. The BIG PRINT giveth and the small print taketh away. You will find a lot of small print. –  IrishStat Jul 24 '12 at 22:17
Totally agree. Fortunately, in time series, it's easy to set up an evaluation mechanism. That's the advice i was trying to convey above. Try several approaches whilst keeping a percentage of your observations tucked away. Then use the different models you have to forecast those observations you kept away. Evaluate the quality of the different models by how good they forecasted the data they didn't see. It's often an illuminating exercise. –  user603 Jul 24 '12 at 22:29
@user603 thanks for your insightful comment. –  Michael Chernick Jul 24 '12 at 22:36
@user603 The problem I see in your suggestion , although "reasonable" is that it is a "single origin' conclusion/test. Time series models/forecasts can often be "incorrect" from a single origin. One needs to evaluate models/forecasts from many origins so I would suggest that this be done from each and every point in the set of "tucked away" values. Recall that a single swallow does not make a summer ! –  IrishStat Jul 24 '12 at 22:58

If you know a priori that a major event occurred at a specific point in time, then one could use a piece of software (that I won't mention because I blush easily!) and specify that this "cause variable" can have contemporaneous and lag structures, THEN a PDL or ADL can be identified to quantify the contemporaneous and lagged structure. This is known as a Transfer Function Model or as an ARMAX model. If you don't have access to this kind of (not-so-expensive) software, then simply lag the indicator/event variable as many times as you think appropriate and estimate the ARMAX model, which of course would include your ARIMA specification and any other indicators reflecting the impact of pulses, level shifts, seasonal pulses and local time trends that were necessary to yield a Gaussian error structure, a.k.a. white noise.

-
I think the DSE package estimates ARMAX models in R. –  Damien Jul 27 '12 at 22:38