0
$\begingroup$

I am working on building a machine learning model to detect the drift in trend, whether upward or downward trends (see the figure attached). The idea to send alarms when the uptick or down-tick happens and the data reaches a control limit, where the equipment results fatal abort.

I have been looking different methods, but not been successful, for example I tried LSTM anomaly detection, it works if I have already know when the measurements starts drifting, then I can take the earlier data for training and later data for testing. In my current situation, I wouldnt know when the drift happens to split data for training and testing. Moreover, the model should look back for last 2 or 3 days data and see if the trend drifting upward or downward.

Below is the picture with hypothetical data to convey the idea. I greatly appreciate if you point me in the right direction or please share your wisdom on how to go about it.

enter image description here

Thank you

$\endgroup$
3
  • $\begingroup$ ehm... hardcoding it? $\endgroup$
    – carlo
    May 21, 2020 at 16:19
  • 1
    $\begingroup$ So clarify, are you trying to predict when the process will cross the control limit? $\endgroup$ May 21, 2020 at 16:19
  • $\begingroup$ Technically, I dont need to predict when it will cross the control limit, because we have alarms set when it crosses the control limit. I want the model to let me know when there is a drift (up or down), for example looking at last 10 data points it should tell me the signals are drifting up or down. $\endgroup$
    – Vishwas
    May 21, 2020 at 17:37

2 Answers 2

2
$\begingroup$

I had a very similar scenario to this where I needed to flag when a pump's pressure time series exceeded a certain threshold, but the data was unavoidably EXTREMELY noisy. The solution involved several steps, which succeeded in triggering at very sensible locations, and I think the method will work very well for you.

  1. Apply a median filter. That is, for each point, take $n$ points to the right and $n$ points to the left, and the point itself, and replace the middle point with the median of all those values. Note that you want to keep referring to the old values, here, and not allow the updated value at $i,$ say, to affect the median value at $i+1.$ Choosing $n$ will require some trial-and-error on your part. The median filter is terrific at reducing spikes, but it is also essentially a low-pass filter. So your filtered signal will respond more slowly than the raw signal.

  2. Fit a reasonably-high-order polynomial to the data, perhaps in sections (like a cubic spline).

  3. Finally, use the extrapolated values from the polynomial to detect threshold crossings.

$\endgroup$
2
  • $\begingroup$ Thank you, interesting, let me try it out and let you know. Do you have notebook or git hub post to reference it? $\endgroup$
    – Vishwas
    May 26, 2020 at 16:21
  • $\begingroup$ I'm afraid I don't. I've done this once in LabVIEW and once in Excel. I have not done it in Python, although I'm sure you can. $\endgroup$ May 26, 2020 at 17:54
1
$\begingroup$

If you like to trigger an alarm before your measurement is crossing the limit and you want to deploy this in a live setting, you can look at last $N$ timesteps and calculate the trend (or multiple trends on different time scales), e.g by linear regression. Now you can compute how many timesteps it would take to cross the limit and create an alarm if necessary.

To look further back in time and recognize seasonal trends and such you can have a look at autogregressive models which are commonly used for timeseries forecasting, there is a nice python library as well (statsmodels). Facebook Research also developed a nice forecasting model based on deep learning (Prophet), never used it myself though.

$\endgroup$
1
  • $\begingroup$ Thank you for the input, I am yet to try ARIMA and Phrophet. Meanwhile, I have already tried LSTM, which solves similar function. However, these work if I know when the trend starts going up or down and then split the before data for training and later data for testing. But if we dont know when it goes up or goes down, then how to go about it? $\endgroup$
    – Vishwas
    May 21, 2020 at 17:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.