I have real time data source that emits numeric values every 5 seconds. I wanted to raise alert whenever, for example the last 5 consecutive values, deviate more than a certain level. As you can see the figure below, values oscillates between max and min and the peak (max) gradually increases. Ideally when it is at peak, it should remain constant or small up and down is also acceptable. However, sudden or slow consecutive increase or decrease is when I want to raise alert.

I played with different models like AR (autoregressive model) and ARIMA but the sample is below recommended in a given period. On top of that it usually takes long time to predict.

I appreciate greatly if you guide me to any model or approach that is more suitable for this task. I don't want to predict the next value given the current value, I want to check at the peak the past few values are increasing or decreasing in a given period.

   X = data['weight'].values
size = int(len(X) * 0.02)
train, test = X[0:size], X[size:len(X)]
history = [x for x in train]
predictions = list()
for t in range(len(test)):
    model = ARIMA(history, order=(5,1,0))
    model_fit = model.fit(disp=0)
    output = model_fit.forecast()
    yhat = output[0]
    obs = test[t]
    print('predicted=%f, expected=%f' % (yhat, obs))
error = mean_squared_error(test, predictions)
print('Test MSE: %.3f' % error)
# plot
  • $\begingroup$ Best is to model the underlying mechanism. What produces these curves? Why are they precisely that shape? Once you can answer these types of questions, you can define what deviation really means and you'll be able to set a threshold for your alert. This is a theory-driven approach as opposed to data-driven methods like ARIMA. $\endgroup$ – LBogaardt May 23 at 16:57
  • $\begingroup$ Additionally, does your alert need to be in real time, so as the stream of values come in? Or do you have a complete dataset ready and do you want to highlight the deviations in retrospect as you have done on your image? $\endgroup$ – LBogaardt May 23 at 17:02
  • $\begingroup$ Thanks @LBogaardt, the curve is produced by weight measuring sensor, so the shape of the curve is when you put weight lift it up and hold a bit before the weight is taken off ,rinse repeat. The alert or notifications should be in real time as well. Actually the most relevant data is within minimum to maximum and back again to minimum cycle. So the deviation I am interested within this period. What happened in the previous cycle is not important. $\endgroup$ – Dolarious May 24 at 9:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.