# How to characterize abrupt change?

This question may be too basic. For a temporal trend of a data, I would like to find out the point where "abrupt" change happens. For example, in the first figure shown below, I would like to find out the change point using some statistic method. And I would like to apply such method in some other data of which the change point is not obvious (like the 2nd figure).So is there a common method for such purpose?

• the term "turning point" has a particular meaning that I don't think applies to a sudden shift in level (whether up or down). You also use the phrase 'change point', and I think that's probably a better choice. Please don't think this is 'too basic'; even basic questions are welcome with no need for apology, and this question isn't remotely basic. – Glen_b Sep 22 '14 at 22:14
• Thanks. I have changed the 'turning point' to 'change point' in the question. – user2230101 Sep 23 '14 at 12:44

If the observations of your time series data are correlated with the immediately previous observations, the paper by Chen and Liu (1993)$^{[1]}$ may interest you. It describes a method to detect level shifts and temporary changes in the framework of autoregressive moving-average time series models.

[1]: Chen, C. and Liu, L-M. (1993),
"Joint Estimation of Model Parameters and Outlier Effects in Time Series,"
Journal of the American Statistical Association, 88:421, 284-297

• +1 I was trying (but failing) to remember enough about this paper to locate it. It's a good reference. – Glen_b Sep 23 '14 at 9:07

This problem in Stats is referred to as the (univariate) Temporal Event Detection. The simplest idea is to use a moving average and standard deviation. Any reading that is "out of" 3-standard deviations (rule-of-thumb) is considered an "event". There of course, are more advanced models that use HMMs, or Regression. Here is an introductory overview of the field.

• This is the only post publicly accessible on the entire Web to include the phrase "Univariate Temporal Event Detection"! What is your source for this term? – whuber Sep 22 '14 at 20:49
• Sorry if it was confusing. Event Detection is a more common term, and Temporal is sometimes used separately. Univariate is not commonly used as the approaches are typically multivariate, but it is his special case. – user1669710 Sep 22 '14 at 22:07
• edited the answer to incorporate your comment @whuber – user1669710 Sep 22 '14 at 22:09
• @ser1669710 Thanks. This is what I am looking for. Seems that moving average cannot solve my problem. I need to look at the more complicated model. – user2230101 Sep 23 '14 at 12:59
• I would like to learn more about this temporal event detection. The slides you posted are nice, but I was wondering if you have the link to a review paper that describes the field a bit more formally? – aaragon Aug 14 '15 at 1:20

This inference problem has many names, including change points, switch points, break points, broken line regression, broken stick regression, bilinear regression, piecewise linear regression, local linear regression, segmented regression, and discontinuity models.

Here is an overview of change point packages with pros/cons and worked examples. If you know the number of change points a priori, check out the mcp package. First, let's simulate the data:

df = data.frame(x = seq(1, 12, by = 0.1))
df$y = c(rnorm(21, 0, 5), rnorm(80, 180, 5), rnorm(10, 20, 5))  For your first problem, it's three intercept-only segments: model = list( y ~ 1, # Intercept ~ 1, # etc... ~ 1 ) library(mcp) fit = mcp(model, df, par_x = "x")  We can plot the resulting fit: plot(fit)  Here, the change points are very well defined (narrow). Let's summarise the fit to see their inferred locations (cp_1 and cp_2): summary(fit) Family: gaussian(link = 'identity') Iterations: 9000 from 3 chains. Segments: 1: y ~ 1 2: y ~ 1 ~ 1 3: y ~ 1 ~ 1 Population-level parameters: name mean lower upper Rhat n.eff cp_1 3.05 3.0 3.1 1 6445 cp_2 11.05 11.0 11.1 1 6401 int_1 0.14 -1.9 2.1 1 5979 int_2 179.86 178.8 180.9 1 6659 int_3 22.76 19.8 25.5 1 5906 sigma_1 4.68 4.1 5.3 1 5282  You can do much more complicated models with mcp, including modeling Nth-order autoregression (useful for time series), etc. Disclaimer: I am the developer of mcp. Here is a quick and easy way to do it. Create a bunch of jump functions like this: $$J_i = \left\{\begin{array}{l@{\qquad}l} 0 & x < x_i\\ 1 & x \ge x_i \end{array}\right.$$ for candidate cutoff points$x_1<x_2<\cdots<x_m$. Now use stepwise regression to select the best model with the$J_i$as possible predictors. In your first example, assuming you select two predictors, you'll get one for$J_{april}$with a positive coefficient equal to the size of the jump upward, and one for$J_{december}$with a negative coefficient equal to the size of the jump downward. You need to decide how finely you want to divide the candidate jump times,$x_i$, e.g., one per month, one per fortnight, one per week, one per day. There are more elegant and exacting solutions involving nonlinear regression, where you use a model with$J_1$and$J_2$and estimate$x_1$and$x_2\$ as parameters. It's a bit messy to set up.

• PS - @user1669710 and I posted answers simultaneously. I voted for that one because it is obviously better researched. But I'm leaving this here as it is an alternative that does work and is easy to implement. – Russ Lenth Sep 22 '14 at 20:31
• Because it uses stepwise regression and employs many candidate variables, this procedure looks suspect. Where has it been studied and what properties does it have? How does it compare to other changepoint methods? – whuber Sep 22 '14 at 20:51
• @whuber, my point exactly. That's why I voted for the other answer. It won't compare too favorably if you have a very granular set of changepoint values. And it might not even compare that favorably otherwise. I'm just putting it out as an ad hoc method, and I think I presented it as such. But I think a method like this promises to be a good way to obtain starting values for the nonlinear method. – Russ Lenth Sep 22 '14 at 21:07
• The idea underlies some of the more effective changepoint methods I have found, but the use of stepwise regression in particular makes me suspect (although I am unsure) that this method could fail even to produce reasonable starting points for other methods to improve on. That is why I am curious whether it has even been studied. – whuber Sep 22 '14 at 21:11
• I think there would few problems with all-subsets selection, as long as there really are a specified number of jumps (say two), as we'd find the two jumps that best explain the data. Other selection methods could be problematic, just as they are in other situations. I think it depends on how important it is to get the best answer, versus a good answer, versus a quick answer. Not all problems are the same, nor are all clients. The best answer in the world is an utter failure if you can't explain it. – Russ Lenth Sep 23 '14 at 0:49

There is a related problem of dividing a series or sequence into spells with ideally constant values. See How can I group numerical data into naturally forming "brackets"? (e.g. income)

It's not quite the same problem as the question doesn't exclude spells with slow drift in any or all directions, but without abrupt changes.

A more direct answer is to say that we are looking for big jumps, so the only real issue is to define jump. The first idea is then just to look at first differences between neighbouring values. It's not even clear that you need to refine that by removing noise first, as if jumps can't be distinguished from differences in noise, they surely can't be abrupt. On the other hand, the questioner evidently wants abrupt change to include ramped as well as stepped change, so some criterion such as variance or range within fixed-length windows seems called for.

The area of statistics that you are looking for is changepoint analysis. There is a website here that will give you an overview of the area and also have a page for software.

If you are an R user then i'd recommend the changepoint package for changes in mean and the strucchange package for changes in regression. If you want to be Bayesian then the bcp package is good too.

In general you have to choose a threshold which indicates the strength of the changes you are looking for. There are, of course, threshold choices that people advocate in certain situations and you can use asymptotic confidence levels or bootstrapping to get confidence too.

• The OP identified two examples, one of which I would call a step and the other a ramp, although there is always scope for wrangling about words. See also my answer here. How do these methods cope with ramps? Do they have a tacit or explicit model of stepped change? – Nick Cox Oct 10 '14 at 8:52
• Thanks for the question Nick. Generally it depends on how long the ramp is. If it is a short ramp then it is treated as 1 change, if the ramp is longer then often the changepoint methods will identify 2 changes, 1 at the start of the ramp and 1 at the end. Obviously this does depend on the underlying model that you assume. – adunaic Oct 13 '14 at 13:35