8
$\begingroup$

I'm a beginner using machine learning (I finished Ng's course), I'm using scikit-learn in python. I want to find the best way to detect anomalies in our system.

We have ongoing events that occur at a schedule (every few min/hours), and I want to detect when something abnormal happens. Example data:

ID | epoch-time | duration (Sec) | status | is_manual

0400 | 1488801454  | 500 | completed | 1

0401 | 1488805055  | 500 | completed | 1

0402 |  1488812254  | 40000 | failed | 1

6831 | 1488805050  | 200 | failed | 0

.

... (Millions of examples)

.

0014 |  1488805055 | 1200 | completed | 0

so for example event ID 0400 occurs once every hour. I want to tell when it does not run.

What I plan to do is feed the algorithm all the events from the last 10 minutes.

Main questions: How to treat the ID column? What is the best approach I should take?

$\endgroup$
  • $\begingroup$ Is anomaly the same as an outlier? In statistics we use the term outlier while in space science they call it an anomaly. $\endgroup$ – Michael Chernick Mar 6 '17 at 15:23
  • $\begingroup$ Difference between outliner and anomaly: stats.stackexchange.com/questions/189664/… $\endgroup$ – XOmri Mar 7 '17 at 12:15
  • $\begingroup$ Can you elaborate a bit more on what an event in this case is? How many unique events are roughly in your data? Also, you gave an example in which the 'abnormal' behavior was the event failing. Are there other cases that you'd see as abnormal? $\endgroup$ – deemel Mar 20 '18 at 10:44
  • $\begingroup$ @Rickyfox By event I mean a row, or input. It was wrong to use the same ID for different events, and I fixed it in the question. Abnormal would be the case where an event is not consistent with the previous events that correlates based on the time. For example: If every 30 seconds, an event occurring with the same parameters (duration: 500, completed, 1), then if there was no event after 30 seconds, that's abnormal. Or if it's failed and not completed: It's also an anomaly. $\endgroup$ – XOmri Mar 21 '18 at 16:11
  • $\begingroup$ I don't see how you need ML here. It seems like a simple conditional query should work just fine $\endgroup$ – Aksakal Mar 21 '18 at 16:30
1
$\begingroup$

I found this article to be very helpful in my case:

https://mapr.com/blog/deep-learning-tensorflow/

Using this basic RNN structure, I was able to predict the outcome of the next timestep. By centering all events to the nearest minute, the network was able to recognize the pattern that correlates within the timeline.

$\endgroup$
-4
$\begingroup$

There are several ways with which you can tackle this. Before jumping into designing any models standardize your data. Your data seems unlabeled, so initially, what you can do is perform a t-SNE visualization on it which will give you a lot insights to your data. Based on its result you can develop more sensible models which can group the samples into normal ones and anomalies. More on t-SNE here

$\endgroup$
  • 2
    $\begingroup$ Welcome to stats.SE! The help center has some good information about asking/answering questions. Detailed answers tend to be best. Can you elaborate a bit more? For example, how would t-SNE be used with time series data and discrete 'id' inputs, and how would it be used to help design an anomaly detection system? $\endgroup$ – user20160 Mar 6 '17 at 14:40
  • $\begingroup$ I'm looking into visualising the data with t-SNE per your suggestion, but I'm not sure how far will I get with it. We have several more features to add that I didn't mention, I will edit and add to the post. I still can't figure out what to do once I get visualization working $\endgroup$ – XOmri Mar 6 '17 at 18:08
  • $\begingroup$ Moreover, t-SNE highly depends on hyperparameters and the does not preserve distances, so how exactly would you find outliers based on it..? Yes it would let you find strange points, but this would be cherry picking. $\endgroup$ – Tim Nov 28 '18 at 8:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.