How to find anomalies in data by using ARIMA

Question

I want to detect anomalies in my data. It is in the red square shown in the figure.

This is a person's chest movement data in 60 seconds. The red square is called "apnea", which a person stops breathing for a short period of time. I was trying to use ARIMA model to detect anomalies. But looks like it is predicting "too good" that it cannot see anomalies.

The ARIMA result and predictions are shown below. Is there any suggestion that I can improve my model so that I can detect anomalies?

EDIT

Raw data in csv is here

Answer 1

post your data in a csv file and I will try and help you by example using Intervention Detection as detailed by Tsay http://docplayer.net/12080848-Outliers-level-shifts-and-variance-changes-in-time-series.html.

EDITED AFTER RECEIPT OF YOUR DATA: You posted 60 values which are in spirit to the data that you presented in this question namely a period of reduced variability signalling the need for possible remedial action. What you are looking for is a collection of sequential values that have a significant variability change.

Your anomalies are BOTH in the Expected Value (first moment periods 6,9,40,10,44) and the Model Error Variance (second moment) period 23-29 in the original data .

@vasilis-vasileiou posted a very intuitive and well-thought out reflection (but very ad hoc ) on how one might analyze and detect the "unusual VARIANCE activity" in your data set. What I propose here is similar in intent and purpose with a little more rigor, so here goes.

To detect pulse anomalies ( changes in the expected value ) and change points in the model error variance change points , one needs to initially condition on any identifiable arima structure which might obfuscate the analysis. In this case there is a strong arima model and some one-time only anomalies caused by some unknown factor. If you don't filter the observed series to create a cleansed version, subsequent steps may not be fruitful.

I used AUTOBOX (which I have helped to develop) and a programmable search algorithm (using 56 residuals) to form some conclusions viz the 6 periods between 19th point through the 25th point (inclusive) was found to have a significant variance change breakpoint suggesting an early warning . Essentially this means that for all startpoints in the series from period 6 through period 50 , we have found that subgroup (6 values ) that has the most significant variance reduction.

Initially the data is here with acf here suggesting a (0,0,0)(1,0,0)4 arima model with a few (5) pulses (change points in the expected value ).

Here is the model and here .

The Cleansed versus the Actual clearly presenting the unusual pulses .

Here is the acf of the model residuals which suggest sufficiency ...

Here are the adjusted Y's i.e. the model residuals which we will further examine for possible structural breakpoints in the error variance . which is now visually obvious.

Note that TSAY suggested tests for latent deterministic structure and non-constancy of model error variance but the error variance test is a before vs after some unknown point in time here http://docplayer.net/12080848-Outliers-level-shifts-and-variance-changes-in-time-series.html .

Unfortunately his test ( available in standard release ) is not suitable to testing when and if the model error variance changed and returned some unknown # of periods in the future to a normative vslue as is the case in your data set.

Taking the vector of model residuals and utilizing a proprietary optimization add-on to AUTOBOX we found that the error variance for periods 19-25 (of 56) was statistically significantly different than elsewhere (p value =.000023 ) which I believe answers your question.

Here is a listing of the residuals (all 60) and a pointer to the targeted subgroup and another picture of the residuals supporting the findings.

This pattern of "small residual variance" is also obvious in the original data set BUT model assumptions and testing are all focused on the residuals from a useful model.

I can imagine this solution being implemented in an Apnea device to warn the user that something might be imminent.

Answer 2

It seems that we need to identify points in time where the time series "doesn't fluctuate that much" (I've put it into quotations because the how much depends on how many false positives etc we allow ourselves to have)

Furthermore, I see from your model that you fitted and AR(3). Thus, I will take the standard deviation of the last 3 observation for each step. I am open to recommendations here but I just thought that I had to draw the line somewhere and last 3 would make sense as the 4th lag wouldn't correlate with the current one.

In R:

st_dev   <- c()

for (i in seq(0, length(x), by = 3)){

  start_point <- i + 1
  stop_point  <- start_point + 2

  if (start_point < length(x)){

    st_dev <- c(st_dev, sd(x[start_point:stop_point]))

  }

}

plot(st_dev)

And afterwards you can apply a rule that says "if 1 of the standard deviation points is below a threshold (can be identified after the person's average measurements or I guess from other studies) then a small alarm. If two consecutive ones are below that threshold then yellow alarm and if three then red alarm."

This is just an idea and an example based on the data you've shared but you can find more generic rules similar to the above idea that would fit the apnea cases.