17
$\begingroup$

I would like to detect changes in time series data, which usually has the same shape. So far I've worked with the changepoint package for R and the cpt.mean(), cpt.var() and cpt.meanvar() functions. cpt.mean() with the PELT method works well when the data usually stays on one level. However I would also like to detect changes during descents. An example for a change, I would like to detect, is the section where the black curve suddenly drops while it actually should follow the examplary red dotted line. I've experimented with the cpt.var() function, however I couldn't get good results. Have you got any recommendations (those don't have to necessarily use R)?

Change curve

Here is the data with the change (as R object):

dat.change <- c(12.013995263488, 11.8460207231808, 11.2845153487846, 11.7884417180764, 
11.6865425802022, 11.4703118125303, 11.4677576899063, 11.0227199625084, 
11.274775836817, 11.03073498338, 10.7771805591742, 10.7383206158923, 
10.5847230134625, 10.2479315651441, 10.4196381241735, 10.467607842288, 
10.3682422713283, 9.7834431752935, 9.76649842404295, 9.78257968297228, 
9.87817694914062, 9.3449034905713, 9.56400153361727, 9.78120084558148, 
9.3445162813738, 9.36767436354887, 9.12070987223648, 9.21909859069157, 
8.85136359917466, 8.8814423003979, 8.61830163359642, 8.44796977628488, 
8.06957847272046, 8.37999165387824, 7.98213210294954, 8.21977468333673, 
7.683960439316, 7.73213584532496, 7.98956476021092, 7.83036046746187, 
7.64496198988985, 4.49693528397253, 6.3459274845112, 5.86993447552116, 
4.58301192892403, 5.63419551523625, 6.67847511602895, 7.2005344054883, 
5.54970477623895, 6.00011922569104, 6.882667104467, 4.74057284230894, 
6.2140437333397, 6.18511450451019, 5.83973575417525, 6.57271194428385, 
5.36261938326723, 5.48948831338016, 4.93968645996861, 4.52598133247377, 
4.56372558828803, 5.74515428123725, 5.45931581984165, 5.58701112949141, 
6.00585679276365, 5.41639695946931, 4.55361875158434, 6.23720558202826, 
6.19433060301002, 5.82989415940829, 5.69321394985076, 5.53585871082265, 
5.42684812413063, 5.80887522466946, 5.56660158483312, 5.7284521523444, 
5.25425775891636, 5.4227645808924, 5.34778016248718, 5.07084809927736, 
5.324066161355, 5.03526881241705, 5.17387528516352, 5.29864121433813, 
5.36894461582415, 5.07436929444317, 4.80619983525015, 4.42858947882894, 
4.33623051506001, 4.33481791951228, 4.38041031792294, 3.90012900415342, 
4.04262777674943, 4.34383842876647, 4.36984816425014, 4.11641092254315, 
3.83985887104645, 3.81813419810962, 3.85174630901311, 3.66434598962311, 
3.4281724860426, 2.99726515704766, 2.96694634792395, 2.94003031547181, 
3.20892607367132, 3.03980832743458, 2.85952185077593, 2.70595278908964, 
2.50931109659839, 2.1912274016859)
$\endgroup$
  • $\begingroup$ Note that if you are only asking for R code, that would be off-topic here. If you are asking for general methodological advice, that is fine. It might come w/ some R code, but then again, it might not. $\endgroup$ – gung Feb 27 '15 at 21:20
  • 1
    $\begingroup$ Good remark, I'm interested in a general solution, using R would just be convenient. $\endgroup$ – mlee Feb 27 '15 at 23:39
17
$\begingroup$

You could use time series outlier detection to detect changes in time series. Tsay's or Chen and Liu's procedures are popular time series outlier detection methods . See my earlier question on this site.

R's tsoutlier package uses Chen and Liu's method for detection outliers. SAS/SPSS/Autobox can also do this. See below for the R code to detect changes in time series.

library("tsoutliers")
dat.ts<- ts(dat.change,frequency=1)
data.ts.outliers <- tso(dat.ts)
data.ts.outliers
plot(data.ts.outliers)

tso function in tsoultlier package identifies following outliers. You can read documentation to find out the type of outliers.

Outliers:
  type ind time coefhat   tstat
1   TC  42   42 -2.9462 -10.068
2   AO  43   43  1.0733   4.322
3   AO  45   45 -1.2113  -4.849
4   TC  47   47  1.0143   3.387
5   AO  51   51  0.9002   3.433
6   AO  52   52 -1.3455  -5.165
7   AO  56   56  0.9074   3.710
8   LS  62   62  1.1284   3.717
9   AO  67   67 -1.3503  -5.502

the package also provides nice plots. see below. The plot shows where the outliers are and also what would have happened if there were no outliers.

enter image description here

I have also used R package called strucchange to detect level shifts. As an example on your data

library("strucchange")
breakpoints(dat.ts~1)

The program correctly identifies breakpoints or structural changes.

Optimal 4-segment partition: 

Call:
breakpoints.formula(formula = dat.ts ~ 1)

Breakpoints at observation number:
17 41 87 

Corresponding to breakdates:
17 41 87 

Hope this helps

$\endgroup$
  • 1
    $\begingroup$ Thanks, tso works well, however it is a little bit slow for larger datasets. The breakpoint positions of struccchange seem a little bit arbitrary (except position 41). $\endgroup$ – mlee Mar 5 '15 at 17:17
7
$\begingroup$

I would approach this problem from the following perspectives. These are just some ideas off the top of my head - please take them with a grain of salt. Nevertheless, I hope that this will be useful.

  • Time series clustering. For example, by using popular dynamic time warping (DTW) or alternative approaches. Please see my related answers: on DTW for classification/clustering and on DTW or alternatives for uneven time series. The idea is to cluster time series into categories "normal" and "abnormal" (or similar).

  • Entropy measures. See my relevant answer on time series entropy measures. The idea is to determine entropy of a "normal" time series and then compare it with other time series (this idea has an assumption of an entropy deviation in case of deviation from "normality").

  • Anomaly detection. See my relevant answer on anomaly detection (includes R resources). The idea is to directly detect anomalies via various methods (please see references). Early Warning Signals (EWS) Toolbox and R package earlywarnings seem especially promising.

$\endgroup$
6
$\begingroup$

My response using AUTOBOX is quite similar to @forecaster but with a much simpler model. Box and Einstein and others have reflected on keeping solutions simple but not too simple. The model that was automatically developed was enter image description here . The actual and cleansed plot is very similar enter image description here . A plot of the residuals (which should always be shown ) is here enter image description here along with the mandatory acf of the residuals enter image description here . The statistics of the residuals are always useful in making comparisons between "dueling models" enter image description here . The Actual/Fit/Forecast graph is hereenter image description here

$\endgroup$
1
$\begingroup$

It would seem that your problem would be greatly simplified if you detrended your data. It appears to decline linearly. Once you detrend the data, you could apply a wide variety of tests for non-stationarity.

$\endgroup$
  • 3
    $\begingroup$ This approach will fail because there are clearly different slopes in the history. Unless you are incorporating multiple "trends/slopes" this approach will not produce meaningful results. Simple straight-forward solutions are often just too simple. $\endgroup$ – IrishStat Feb 28 '15 at 15:43
1
$\begingroup$

All fine answers, but here is a simple one, as suggested by @MrMeritology, which appears to work well for the time series in question, and likely for many other "similar" data sets.

Here is an R-snippet producing the self-explanatory graphs below.

outl = rep( NA, length(dat.change))
detr = c( 0, diff( dat.change))

ix = abs(detr) > 2*IQR( detr)
outl[ix] = dat.change[ix]

plot( dat.change, t='l', lwd=2, main="dat.change TS")
points( outl, col=2, pch=18)

plot( detr, col=4, main="detrended TS", t='l', lwd=2 )
acf( detr, main="ACF of detrended TS")

enter image description here enter image description here enter image description here

$\endgroup$
  • $\begingroup$ there can be multiple trend changes and multiple intercept changes (level shifts) ... thus one needs to find solutions that actually diagnose the data to determine these ... $\endgroup$ – IrishStat Mar 5 at 21:56
  • $\begingroup$ Yes, indeed, I've read your previous comment above. However, diagnosing the time series to detect multiple trends / levels is a problem in itself. My point here is to show that the above simple approach works sometimes, in particular for the given data. Conversely, no single approach will work well always. An approach by R.Hyndman (R-function tsoutliers) is something that I'd otherwise recommend. $\endgroup$ – dnqxt Mar 5 at 22:09
  • $\begingroup$ AUTOBOX is the single approach that will work well always ( at least for the zillions of time series we have seen ) and there is an R version. If you wish to chat offline as I don't want to get to "salesy " here I can explain the process which is fully understandable/transparent but not easily duplicated. $\endgroup$ – IrishStat Mar 5 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.