26
$\begingroup$

I would like to detect changes in time series data, which usually has the same shape. So far I've worked with the changepoint package for R and the cpt.mean(), cpt.var() and cpt.meanvar() functions. cpt.mean() with the PELT method works well when the data usually stays on one level. However I would also like to detect changes during descents. An example for a change, I would like to detect, is the section where the black curve suddenly drops while it actually should follow the examplary red dotted line. I've experimented with the cpt.var() function, however I couldn't get good results. Have you got any recommendations (those don't have to necessarily use R)?

Change curve

Here is the data with the change (as R object):

dat.change <- c(12.013995263488, 11.8460207231808, 11.2845153487846, 11.7884417180764, 
11.6865425802022, 11.4703118125303, 11.4677576899063, 11.0227199625084, 
11.274775836817, 11.03073498338, 10.7771805591742, 10.7383206158923, 
10.5847230134625, 10.2479315651441, 10.4196381241735, 10.467607842288, 
10.3682422713283, 9.7834431752935, 9.76649842404295, 9.78257968297228, 
9.87817694914062, 9.3449034905713, 9.56400153361727, 9.78120084558148, 
9.3445162813738, 9.36767436354887, 9.12070987223648, 9.21909859069157, 
8.85136359917466, 8.8814423003979, 8.61830163359642, 8.44796977628488, 
8.06957847272046, 8.37999165387824, 7.98213210294954, 8.21977468333673, 
7.683960439316, 7.73213584532496, 7.98956476021092, 7.83036046746187, 
7.64496198988985, 4.49693528397253, 6.3459274845112, 5.86993447552116, 
4.58301192892403, 5.63419551523625, 6.67847511602895, 7.2005344054883, 
5.54970477623895, 6.00011922569104, 6.882667104467, 4.74057284230894, 
6.2140437333397, 6.18511450451019, 5.83973575417525, 6.57271194428385, 
5.36261938326723, 5.48948831338016, 4.93968645996861, 4.52598133247377, 
4.56372558828803, 5.74515428123725, 5.45931581984165, 5.58701112949141, 
6.00585679276365, 5.41639695946931, 4.55361875158434, 6.23720558202826, 
6.19433060301002, 5.82989415940829, 5.69321394985076, 5.53585871082265, 
5.42684812413063, 5.80887522466946, 5.56660158483312, 5.7284521523444, 
5.25425775891636, 5.4227645808924, 5.34778016248718, 5.07084809927736, 
5.324066161355, 5.03526881241705, 5.17387528516352, 5.29864121433813, 
5.36894461582415, 5.07436929444317, 4.80619983525015, 4.42858947882894, 
4.33623051506001, 4.33481791951228, 4.38041031792294, 3.90012900415342, 
4.04262777674943, 4.34383842876647, 4.36984816425014, 4.11641092254315, 
3.83985887104645, 3.81813419810962, 3.85174630901311, 3.66434598962311, 
3.4281724860426, 2.99726515704766, 2.96694634792395, 2.94003031547181, 
3.20892607367132, 3.03980832743458, 2.85952185077593, 2.70595278908964, 
2.50931109659839, 2.1912274016859)
Improve this question
$\endgroup$
2
  • $\begingroup$ Note that if you are only asking for R code, that would be off-topic here. If you are asking for general methodological advice, that is fine. It might come w/ some R code, but then again, it might not. $\endgroup$
    – gung - Reinstate Monica
    Commented Feb 27, 2015 at 21:20
  • 1
    $\begingroup$ Good remark, I'm interested in a general solution, using R would just be convenient. $\endgroup$
    – mlee
    Commented Feb 27, 2015 at 23:39

6 Answers 6

Reset to default
20
$\begingroup$

You could use time series outlier detection to detect changes in time series. Tsay's or Chen and Liu's procedures are popular time series outlier detection methods . See my earlier question on this site.

R's tsoutlier package uses Chen and Liu's method for detection outliers. SAS/SPSS/Autobox can also do this. See below for the R code to detect changes in time series.

library("tsoutliers")
dat.ts<- ts(dat.change,frequency=1)
data.ts.outliers <- tso(dat.ts)
data.ts.outliers
plot(data.ts.outliers)

tso function in tsoultlier package identifies following outliers. You can read documentation to find out the type of outliers.

Outliers:
  type ind time coefhat   tstat
1   TC  42   42 -2.9462 -10.068
2   AO  43   43  1.0733   4.322
3   AO  45   45 -1.2113  -4.849
4   TC  47   47  1.0143   3.387
5   AO  51   51  0.9002   3.433
6   AO  52   52 -1.3455  -5.165
7   AO  56   56  0.9074   3.710
8   LS  62   62  1.1284   3.717
9   AO  67   67 -1.3503  -5.502

the package also provides nice plots. see below. The plot shows where the outliers are and also what would have happened if there were no outliers.

enter image description here

I have also used R package called strucchange to detect level shifts. As an example on your data 

library("strucchange")
breakpoints(dat.ts~1)

The program correctly identifies breakpoints or structural changes.

Optimal 4-segment partition: 

Call:
breakpoints.formula(formula = dat.ts ~ 1)

Breakpoints at observation number:
17 41 87 

Corresponding to breakdates:
17 41 87

Hope this helps

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thanks, tso works well, however it is a little bit slow for larger datasets. The breakpoint positions of struccchange seem a little bit arbitrary (except position 41). $\endgroup$
    – mlee
    Commented Mar 5, 2015 at 17:17
8
$\begingroup$

My response using AUTOBOX is quite similar to @forecaster but with a much simpler model. Box and Einstein and others have reflected on keeping solutions simple but not too simple. The model that was automatically developed was enter image description here . The actual and cleansed plot is very similar enter image description here . A plot of the residuals (which should always be shown ) is here enter image description here along with the mandatory acf of the residuals enter image description here . The statistics of the residuals are always useful in making comparisons between "dueling models" enter image description here . The Actual/Fit/Forecast graph is hereenter image description here

Improve this answer
$\endgroup$
0
7
$\begingroup$

I would approach this problem from the following perspectives. These are just some ideas off the top of my head - please take them with a grain of salt. Nevertheless, I hope that this will be useful.

  • Time series clustering. For example, by using popular dynamic time warping (DTW) or alternative approaches. Please see my related answers: on DTW for classification/clustering and on DTW or alternatives for uneven time series. The idea is to cluster time series into categories "normal" and "abnormal" (or similar).

  • Entropy measures. See my relevant answer on time series entropy measures. The idea is to determine entropy of a "normal" time series and then compare it with other time series (this idea has an assumption of an entropy deviation in case of deviation from "normality").

  • Anomaly detection. See my relevant answer on anomaly detection (includes R resources). The idea is to directly detect anomalies via various methods (please see references). Early Warning Signals (EWS) Toolbox and R package earlywarnings seem especially promising.

Improve this answer
$\endgroup$
2
$\begingroup$

Lots of excellent answers are given here. Apparently, the results will depend largely on the models chosen. With that said, allow me to throw one more possibility to this old question based on a Bayesian time series decomposition model I developed, available in an R package Rbeast (https://github.com/zhaokg/Rbeast). It assumes a model of the form Y= seasonality+trend+outliers+error; for each component, changepoints are also detected, if any. Here is a test on the given example:

 library(Rbeast)
 out = beast(dat.change, season='none', ocp=8) # no seasonality with a max of 8 outlier points
# out= beast(dat.change, season='none')        # no outlier component is assumed
plot(out)

Below, the Pr(tcp) subplot gives the probability of having sudden changes (i.e., changepoint) over time.

enter image description here

bcp is another great R package for Bayesian change detection.

Improve this answer
$\endgroup$
1
$\begingroup$

It would seem that your problem would be greatly simplified if you detrended your data. It appears to decline linearly. Once you detrend the data, you could apply a wide variety of tests for non-stationarity.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ This approach will fail because there are clearly different slopes in the history. Unless you are incorporating multiple "trends/slopes" this approach will not produce meaningful results. Simple straight-forward solutions are often just too simple. $\endgroup$
    – IrishStat
    Commented Feb 28, 2015 at 15:43
1
$\begingroup$

All fine answers, but here is a simple one, as suggested by @MrMeritology, which appears to work well for the time series in question, and likely for many other "similar" data sets.

Here is an R-snippet producing the self-explanatory graphs below.

outl = rep( NA, length(dat.change))
detr = c( 0, diff( dat.change))

ix = abs(detr) > 2*IQR( detr)
outl[ix] = dat.change[ix]

plot( dat.change, t='l', lwd=2, main="dat.change TS")
points( outl, col=2, pch=18)

plot( detr, col=4, main="detrended TS", t='l', lwd=2 )
acf( detr, main="ACF of detrended TS")

enter image description here enter image description here enter image description here

Improve this answer
$\endgroup$
3
  • $\begingroup$ there can be multiple trend changes and multiple intercept changes (level shifts) ... thus one needs to find solutions that actually diagnose the data to determine these ... $\endgroup$
    – IrishStat
    Commented Mar 5, 2019 at 21:56
  • $\begingroup$ Yes, indeed, I've read your previous comment above. However, diagnosing the time series to detect multiple trends / levels is a problem in itself. My point here is to show that the above simple approach works sometimes, in particular for the given data. Conversely, no single approach will work well always. An approach by R.Hyndman (R-function tsoutliers) is something that I'd otherwise recommend. $\endgroup$
    – dnqxt
    Commented Mar 5, 2019 at 22:09
  • $\begingroup$ AUTOBOX is the single approach that will work well always ( at least for the zillions of time series we have seen ) and there is an R version. If you wish to chat offline as I don't want to get to "salesy " here I can explain the process which is fully understandable/transparent but not easily duplicated. $\endgroup$
    – IrishStat
    Commented Mar 5, 2019 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.