1
$\begingroup$

I have a 1.5 million row mutivariate time series dataset that looks like this:

Source 1:
+------------------------------------------------------------------+
|  Time    Sensor A    Sensor B    Sensor C    Sensor D    Label   |
+------------------------------------------------------------------+
|  2018-02-12 09:00:00    52.2    152.2    2.12    655.5    0      |
|  2018-02-12 09:02:00    53.3    154.2    2.23    660.8    1      |
|  ...                    ...     ...      ...     ...      ...    |
+------------------------------------------------------------------+
Source 2:
+------------------------------------------------------------------+
|  Time    Sensor A    Sensor B    Sensor C    Sensor D    Label   |
+------------------------------------------------------------------+
|  2018-02-12 09:00:00    52.2    152.2    2.12    655.5    1      |
|  2018-02-12 09:02:00    53.3    154.2    2.23    660.8    1      |
|  ...                    ...     ...      ...     ...      ...    |
+------------------------------------------------------------------+

I plan on stacking all this data from different sources together. I want to learn to detect the label at any time given the input data from the past.

One idea I thought of is to calculate various rolling features like mean, median absolute deviation, iqr, etc. on the data for different rolling window sizes, the using LASSO to get the important features and build a simple logistic regression based model on top of that. This is very computationally intensive since the number of columns would be n_featuresn_windowsn_statistics where n_features is the number of sensors: A - E, n_windows is the number of different rolling window sizes and n_statistics is the number of different statistics that would be calculated (eg. IQR, MAD, STD, etc.). 1. What kind of statistics should I look into? 2. Is this approach any good?

Another is to somehow incorporate an LSTM network that can learn without me explicitly calculating the features. 1. How can I do this? 2. Do you know of any resources that can help with this problem? Most of the stuff I see online is based on forecasting, which this is not.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.