Anomaly detection on time series I'm a beginner using machine learning (I finished Ng's course), I'm using scikit-learn in python.
I want to find the best way to detect anomalies in our system.
We have ongoing events that occur at a schedule (every few min/hours), and I want to detect when something abnormal happens.
Example data:
ID | epoch-time | duration (Sec) | status | is_manual

0400 | 1488801454  | 500 | completed | 1

0401 | 1488805055  | 500 | completed | 1

0402 |  1488812254  | 40000 | failed | 1

6831 | 1488805050  | 200 | failed | 0

.

... (Millions of examples)

.

0014 |  1488805055 | 1200 | completed | 0

so for example event ID 0400 occurs once every hour. I want to tell when it does not run.
What I plan to do is feed the algorithm all the events from the last 10 minutes.
Main questions:
How to treat the ID column?
What is the best approach I should take?
 A: I found this article to be very helpful in my case:
https://mapr.com/blog/deep-learning-tensorflow/ 
Using this basic RNN structure, I was able to predict the outcome of the next timestep. 
By centering all events to the nearest minute, the network was able to recognize the pattern that correlates within the timeline.
A: A simple approach would be to treat each event type as independent, and then to build one model per event type. If you expect events to happen on a regular schedule, then an informative feature could be time-since-last-event.
To evaluate the viability of such a model one should do some Exploratory Data Analysis and plot the histograms of such features and analyze whether outliers are present and visible.
If it looks reasonable then one could fit a model to the features. If the features are continuous and normally distributed, the distance from the normalized distribution might be a decent anomaly score. That is easy to compute for a single feature (univariate). For multiple features one would use something like EllipticEnvelope or GaussianMixtureModel.
One could build a multi-event anomaly scoring model using the anomaly scores fro the per-event models.
I see that there are also failure/complete status for events. One might summarize those over a time-period and compute the (time-averaged) failure-rate. Either per-event-type or across all. This one could also build an anomaly detection model on. Perhaps just simple thresholding.
A: There are several ways with which you can tackle this. Before jumping into designing any models standardize your data. Your data seems unlabeled, so initially, what you can do is perform a t-SNE visualization on it which will give you a lot insights to your data. Based on its result you can develop more sensible models which can group the samples into normal ones and anomalies. More on t-SNE here
