Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 0 down vote favorite

Suppose we have some house price data for 30 years (1970-1999). This is yearly data (30 data points). Suppose some major event $X$ happened on 1980. I want to see whether this event affected prices later on. Is using median filters a good methodology for detecting outliers in later years?

share|improve this question
@user603: Let $y_1 = \{1,2,3,4 \}$ and $y_2 = \{1,2,3,4,10,68 \}$. The median filter for the first would be $\{1,2,3,4 \}$ and the median filter for the second would be $\{1,2,3,4,10,68 \}$. – Damien Jul 24 '12 at 17:29
Meant subsitute i.e. substitute some genuine observations by some outliers and see the result. Also, the #of outliers< T/2-1 (T is the length of your series). – user603 Jul 24 '12 at 17:56
Damien, I don't think you're computing the filter correctly: you need to be more careful at the ends of the series. A median-of-three filter would give $(*,2,3,*)$ and $(*,2,3,4,10,*)$ in those cases, with "$*$" representing a value that shouldn't be trusted. Notice that medians (of 3) give you no defense against pairs of adjacent outliers, either, but wider-window medians will be more stable. – whuber♦ Jul 24 '12 at 18:04

2 Answers

up vote 0 down vote

If you know apriori that a major event occurred at a specific point in time then one could use a piece of software ( that I won't mention beacause I blush easily ! ) and specify that this "cause variable" can have contemporaneous and lag structures THEN a PDL or ADL can be identified to quantify the contemporaneous and lagged structure. This is known as a Transfer Function Model or as an ARMAX model. If you don't have access to this kind of (not-so expensive) software then simply lag the indicator/event variable as many times as you think appropriate and estimate the ARMAX model which of course would include your ARIMA specification and any other indicators reflecting the impact of pulses,level shifts , seasonal pulses and local time trends that were necessary to yield a Gaussian error structure a.k.a. white noise.

share|improve this answer
I think the DSE package estimates ARMAX models in R. – Damien Jul 27 '12 at 22:38
up vote 0 down vote

i like the idea of looking for interventions that form level changes in a process. This can be done using ARIMA models and search for the changepoints. Our friend IrishStat does this with his autobox software as he so nicely demonstated yesterday.

I am sure he will be happy to give you more details and perhaps even do a data analysis for you.

I wouldn't call the level shifts outliers. There are various types of outliers in time series and numerous ways to try to detect them. We also discussed this either yesterday or today. Fox identified two type of outliers in his paper from back in the 70s.

share|improve this answer
There is also a changepoint package in R. – Damien Jul 24 '12 at 19:09
@Damien There is a changepoint package in R. It is important for you to know all the caveats i.e.all the assumptions underlying the tests that are conducted. The BIG PRINT giveth and the small print taketh away. You will find a lot of small print. – IrishStat Jul 24 '12 at 22:17
Totally agree. Fortunately, in time series, it's easy to set up an evaluation mechanism. That's the advice i was trying to convey above. Try several approaches whilst keeping a percentage of your observations tucked away. Then use the different models you have to forecast those observations you kept away. Evaluate the quality of the different models by how good they forecasted the data they didn't see. It's often an illuminating exercise. – user603 Jul 24 '12 at 22:29
@user603 thanks for your insightful comment. – Michael Chernick Jul 24 '12 at 22:36
@user603 The problem I see in your suggestion , although "reasonable" is that it is a "single origin' conclusion/test. Time series models/forecasts can often be "incorrect" from a single origin. One needs to evaluate models/forecasts from many origins so I would suggest that this be done from each and every point in the set of "tucked away" values. Recall that a single swallow does not make a summer ! – IrishStat Jul 24 '12 at 22:58

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.