# Detecting steps in time series

I've attached a picture of the time series I'm talking about. The top is the original series, the bottom is the differenced series.

Each data point is a 5 minute average reading from a strain gauge. This strain gauge is placed on a machine. The noisy areas correspond to areas where the machine is turned on, the clean areas are when the machine is turned off. If you look at the area circled in red, there are anomalous steps in the reading that I would like to be able to detect automatically.

I'm completely stumped on how I might be able to do this - any ideas? • What is anomalous steps? is it the machine is turned off? I could see similar patten in red circle in the other part of series? Did I miss something to see in the plot? – vinux Jan 5 '12 at 5:25
• This seems difficult. My fisrt thought: model the (hidden) on/off state by a Markov Chain. The phenomenon you want to detect is a the occurence of a few important values in the differenced series when the machine is off, so if you can compute at each moment the probability for the machine to be off, you may be able detect it as "important values when the off probability is high". – Elvis Jan 5 '12 at 7:10
• I had a second thought. I’ll post it... – Elvis Jan 5 '12 at 9:21

To implement this recipe, we need to choose (a) how close "nearby" means, (b) a recipe for smoothing, and (c) a recipe for finding local variation. You may have to experiment with (a), so let's make it an easily controllable parameter. Good, readily available choices for (b) and (c) are Lowess and the IQR, respectively. Here is an R implementation:
library(zoo)                      # For the local (moving window) IQR
r <- zoo(x - y$y) # Its residuals, structured for the next step z <- rollapply(r, width, IQR) # The running estimate of variability r/z # The diagnostic series: residuals scaled by IQRs }  As an example of its use, consider these simulated data where two successive spikes are added to a quiet period (two in a row should be harder to detect than one isolated spike): > x <- c(rnorm(192, mean=0, sd=1), rnorm(96, mean=0, sd=0.1), rnorm(192, mean=0, sd=1)) > x[240:241] <- c(1,-1) # Add a local spike > plot(x) Here is the diagnostic plot: > u <- f(x) > plot(u) Despite all the noise in the original data, this plot beautifully detects the (relatively small) spikes in the center. Automate the detection by scanning f(x) for largish values (larger than about 5 in absolute value: experiment to see what works best with sample data). > spikes <- u[abs(u) >= 5] 240 241 273 9.274959 -9.586756 6.319956  The spurious detection at time 273 was a random local outlier. You can refine the test to exclude (most) such spurious values by modifying f to look for simultaneously high values of the diagnostic r/z and low values of the running IQR, z. However, although the diagnostic has a universal (unitless) scale and interpretation, the meaning of a "low" IQR depends on the units of the data and has to be determined from experience. • The last paragraph was based on simulated data much like, but not exactly the same as, those shown here. In this diagnostic plot it is clear there are other spurious values up to about 7.5 in size and the two spikes have values around 14. – whuber Jan 5 '12 at 15:27 • Nice answer - this looks promising. Thanks a lot. I'll get back to you later with my results. – mohamedmoussa Jan 5 '12 at 21:34 • Just a quick follow up - I did a similar thing to what you did, but I had access to the standard deviation of the data within each data point (ie. within each 5 min period). I divided by the smoothed STD and I got some pretty good results. – mohamedmoussa Jan 10 '12 at 22:09 • wow, I missed that! (+1!) – Elvis Jan 14 '12 at 13:37 Here is a two cents suggestion. Denote$X_t$the differenced series. Given$\Delta > 0$and a point$t$, define $$a(\Delta,t) = {1\over 2\Delta + 1} |X_t|.$$ For let’s says$\Delta = 50$, the value of$a(\Delta,t)$characterizes the off/on zones by low/high values. An anomalous step is a point$t$where$|X_t| > \alpha a(\Delta,t)$– you’ll need to do some tuning on$\alpha, \Delta$to detect what you want, and avoid false positive when the machine turns on. I’d try first with$\Delta = 50$and$\alpha = 4$. Alternatively, you can look at points$t$where$a(\delta,t) > \alpha a(\Delta,t)$for a$\delta\ll\Delta$(eg$\delta = 10$,$\Delta = 100$), that may help the fine tuning (in that case, you would take a smaller value for$\alpha\$).