2
$\begingroup$

I have a time series of data, which usually is pretty stable (but with noise). Sometimes, it starts to drift linearly. It could also reset back to the stable position and continue the same drift.

Is there an algorithm that can detect these periods of drifts? My naive method would be:

  1. Try to find an period of more than 100 samples where a linear fit could decrease the variation significantly (say R^2 > .8).
  2. Make this period as big as possible.
  3. Continue with #1, but also impose a penalty on detecting too much periods.

Does something like this exists?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
2
$\begingroup$

What you describing is the detection of breakpoints in time series due to changes in deterministic trends. This is referred to as Intervention Detection and was first suggested by I.Chang supervised by G.C.Tiao in 1979 (see https://arxiv.org/pdf/1101.0912.pdf) for more on outlier detection. Early work did not include Trend Detection but more recent software advances does so. The idea is that you don't know a priori how many individual points there are in a new trend or the number of "new trends" . Simple software has approached this with the assumption of the length (duration) of the trends/splines , the # of new trends and naively no ARIMA structure ; all to no avail . Most implementations of Intervention Detection ignore seasonal pulses but do have the facility to detect trend changes which must be done in concert with ARIMA structure. Trying to find trend change points without considering ARIMA structure is like fighting with one hand tied behind your back.

Additionally most implementations of Intervention Detection require pre-specification of the ARIMA structure whereas both components need to be found simultaneously, a neat trick that can be rarely found. Additionally if you have user-suggested predictor variables Intervention Detection is also generally unavailable.

I suggest that you simulate an example or 2 or 3 (not toooo complex) and try different software packages that offer Intervention Detection and see how it goes.

I should add that Intervention Detection was also popularized by William Bell https://www.federalpay.org/employees/bureau-of-the-census/bell-william-r and R. Tsay http://docplayer.net/12080848-Outliers-level-shifts-and-variance-changes-in-time-series.html

You might also search on user:3382 TREND DETECTION as I have commented on trend detection before. Particularly Detecting initial trend or outliers might be illuminating

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the name ("Intervention Detection") and the many references! I'll try to work with those this week. $\endgroup$
    – Frank Meulenaar
    Commented Nov 27, 2016 at 15:00
  • 1
    $\begingroup$ Please see @Richard Hardy excellent description/contrast of stochastic vs deterministic trends stats.stackexchange.com/questions/247977/… $\endgroup$
    – IrishStat
    Commented Nov 28, 2016 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.