# Need advice on change point (step) detection

I have a time series with lots of steps/jumps (data file here). A plot is given below. I would like to subtract an appropriate value for each of these square wave features to bring them back down to the baseline of the signal (i.e. remove the jumps so I get a smoothly varying signal). A median filter works really well for removing a small number of outliers in a row, but in this case I probably need a different approach since the square wave jumps can have different durations as seen. A common method I've seen for doing this is to compute first differences of adjacent samples, and look for large differences to detect jumps. I implemented this method but the problem is it often fails, since the one tunable parameter for the method is a threshold value $$t$$ which the first differences must cross in order to detect a jump: $$| x_{i+1} - x_i | > t$$ As can be seen in the plot below, the jumps I have are often different sizes, so a constant threshold value isn't the best approach. In particular, in some cases there is an interesting signal where adjacent samples can change by large values without being a jump! I have highlighted such a region in red.

Below is a zoomed in view of the red box area. You can see there is a square wave jump followed by an interesting signal. The red arrows depict a place where adjacent samples from an interesting signal have a larger distance between them than some of the jumps in the signal. Therefore a constant threshold method with finite differencing will not work for me.

Does anyone know of a robust procedure to detect and subtract the square wave jumps to end up with a smoothly varying signal with no jumps? I'm sure this must be a solved problem but I haven't had much luck searching online.

This is not exactly an answer, but an attempt to illustrate further the problems I'm having with the first differences method. In the figure below, I focus on that same region as in the original question. I have added a median filtered curve in green (7-point centered window), plus the curve of first differences of the median filtered curve in red ($$x_{i+1}-x_i$$). Finally in blue I plot a constant "threshold" value of $$\pm 4$$ to identify a step. http://dropcanvas.com/d6jsb/1

You can see that the first difference curve (red) identifies the step jump quite well, but around 30:00 the red curve also passes the threshold value when in fact there is no jump there. I cannot increase the threshold value any more because of the following situation a little later in the time series: Here you can see two step features in a row whose first differences don't quite pass the $$\pm 4$$ threshold. So these jumps are not correctly identified. So the question becomes how to "enhance" the first difference signal in the presence of a step, while reducing it in non-step cases like in the first figure above. One idea I have is to compute the median absolute deviation (MAD) of the previous N samples, and compare the first difference value with the MAD instead of using a constant threshold. But I haven't gotten this method to work very well either, because sometimes the MAD is very small, and even a modest first difference value would result in false positive.

An updated data file with the median filter and first difference values is here. I would greatly appreciate any ideas!

• I might be in a very small minority here but I have never understood how one could create a procedure to find oddities (jumps, outliers, etc.) without first defining very specifically as to what an oddity is. In other words, I don't think a criterion of "I'll know it when I see it" is very useful or consistent. So...I think a specific definition for the types of oddities of interest along with a model as to how the data is generated and/or collected is necessary to proceed.
– JimB
Commented Feb 17, 2018 at 21:21
• In this case, I would like to detect and correct square-wave type jumps/steps in my time series. In other words there is an abrupt change in the mean of the time series for some amount of time, and then it returns to the previous baseline.
– vibe
Commented Feb 19, 2018 at 1:52
• reminds me of this thread: stats.stackexchange.com/questions/1142/… Commented Feb 19, 2018 at 2:31
• why don't you just set your threshold to something like $|x_{i+1}-x_i|/S>t$ where $S$ is the sample variance from the previous $m$ (window size, maybe 30) points? And maybe calculate $S$ in such a way that if there are any huge deviations they are not included in the calculation? Commented Jun 3, 2020 at 2:17
• Is there some kind of periodicty in the signal as the first figure suggests?
– Yves
Commented Mar 24, 2021 at 15:54

I don't know what language you are using, but for Matlab I wrote this Shape based filter. It's a "match based filter".
Matlab can be read as psuedo code for other languages.

I wrote it for a sawtooth pattern I needed to remove, but also used it for removing steps. The shapes you are matching (to remove) are step ups and step downs. Let's say [0,0,0,0,1,1,1,1] or [1,1,1,1,0,0,0,0]

You can use a short sliding window of a number of samples that can be parametrized, and compute the range over it. When this window is at the "edge" of a jump, the range statistic will change considerably. Than let A and B be the indices of successive range changes, i.e. jumps. You can push each of the values between these indices by the value of range, i.e. the depth of the jump, and get rid of it.

Another method you can use is to make an index of the range statistic as well as mean, than use the sliding window approach to find anomalies in the range-mean index series.

• Unfortunately, ranges are usually too volatile to be used reliably. A range-based threshold will catch all individual "blips." You can see a few in the examples and it looks the OP is uninterested in those. They usually would not be considered changepoints by anyone.
– whuber
Commented Apr 20, 2023 at 22:30
• @whuber that's one of the reasons why i figured including mean as well would probably be a better way to go, but i guess experimenting with different ideas is necessary in such a problem. What do you suggest? Commented Apr 21, 2023 at 11:56
• As the comments to this question indicate, the problem is not very well defined. There are many changepoint methods available for various kinds of regression. At stats.stackexchange.com/a/377390/919 I describe a simple, well-established method based on windowed medians that could be applied to the residuals from a preliminary robust smooth of these data. Isolated outliers could first be removed using, e.g., a method like stats.stackexchange.com/a/35612/919. A subtler method to detect local changes in variability is given at stats.stackexchange.com/a/20626/919.
– whuber
Commented Apr 21, 2023 at 13:12

From the changepoint detection literature, it seems that SiMUltaneous Changepoint Estimation (SMUCE) approach could be useful.

There is lots of literature, and there is an R package provided by the authors where they provide the estimator for several setups (including autocorrelation in the time series)

The paper Pein et al (2017) explains the theory. An easy implementation in R is available in the stepR package

It will detect all changepoints in your series, that you can later use to remove the episodes that you want.

Good luck!