# Anomaly detection on time series

I'm a beginner using machine learning (I finished Ng's course), I'm using scikit-learn in python. I want to find the best way to detect anomalies in our system.

We have ongoing events that occur at a schedule (every few min/hours), and I want to detect when something abnormal happens. Example data:

ID | epoch-time | duration (Sec) | status | is_manual

0400 | 1488801454  | 500 | completed | 1

0401 | 1488805055  | 500 | completed | 1

0402 |  1488812254  | 40000 | failed | 1

6831 | 1488805050  | 200 | failed | 0

.

... (Millions of examples)

.

0014 |  1488805055 | 1200 | completed | 0


so for example event ID 0400 occurs once every hour. I want to tell when it does not run.

What I plan to do is feed the algorithm all the events from the last 10 minutes.

Main questions: How to treat the ID column? What is the best approach I should take?

• Is anomaly the same as an outlier? In statistics we use the term outlier while in space science they call it an anomaly. – Michael Chernick Mar 6 '17 at 15:23
• Difference between outliner and anomaly: stats.stackexchange.com/questions/189664/… – XOmri Mar 7 '17 at 12:15
• Can you elaborate a bit more on what an event in this case is? How many unique events are roughly in your data? Also, you gave an example in which the 'abnormal' behavior was the event failing. Are there other cases that you'd see as abnormal? – deemel Mar 20 '18 at 10:44
• @Rickyfox By event I mean a row, or input. It was wrong to use the same ID for different events, and I fixed it in the question. Abnormal would be the case where an event is not consistent with the previous events that correlates based on the time. For example: If every 30 seconds, an event occurring with the same parameters (duration: 500, completed, 1), then if there was no event after 30 seconds, that's abnormal. Or if it's failed and not completed: It's also an anomaly. – XOmri Mar 21 '18 at 16:11
• I don't see how you need ML here. It seems like a simple conditional query should work just fine – Aksakal Mar 21 '18 at 16:30