8
$\begingroup$

I'm a beginner using machine learning (I finished Ng's course), I'm using scikit-learn in python. I want to find the best way to detect anomalies in our system.

We have ongoing events that occur at a schedule (every few min/hours), and I want to detect when something abnormal happens. Example data:

ID | epoch-time | duration (Sec) | status | is_manual

0400 | 1488801454  | 500 | completed | 1

0401 | 1488805055  | 500 | completed | 1

0402 |  1488812254  | 40000 | failed | 1

6831 | 1488805050  | 200 | failed | 0

.

... (Millions of examples)

.

0014 |  1488805055 | 1200 | completed | 0

so for example event ID 0400 occurs once every hour. I want to tell when it does not run.

What I plan to do is feed the algorithm all the events from the last 10 minutes.

Main questions: How to treat the ID column? What is the best approach I should take?

Improve this question
$\endgroup$
6
  • $\begingroup$ Is anomaly the same as an outlier? In statistics we use the term outlier while in space science they call it an anomaly. $\endgroup$
    – Michael R. Chernick
    Commented Mar 6, 2017 at 15:23
  • $\begingroup$ Difference between outliner and anomaly: stats.stackexchange.com/questions/189664/… $\endgroup$
    – XOmri
    Commented Mar 7, 2017 at 12:15
  • $\begingroup$ Can you elaborate a bit more on what an event in this case is? How many unique events are roughly in your data? Also, you gave an example in which the 'abnormal' behavior was the event failing. Are there other cases that you'd see as abnormal? $\endgroup$
    – deemel
    Commented Mar 20, 2018 at 10:44
  • $\begingroup$ @Rickyfox By event I mean a row, or input. It was wrong to use the same ID for different events, and I fixed it in the question. Abnormal would be the case where an event is not consistent with the previous events that correlates based on the time. For example: If every 30 seconds, an event occurring with the same parameters (duration: 500, completed, 1), then if there was no event after 30 seconds, that's abnormal. Or if it's failed and not completed: It's also an anomaly. $\endgroup$
    – XOmri
    Commented Mar 21, 2018 at 16:11
  • $\begingroup$ I don't see how you need ML here. It seems like a simple conditional query should work just fine $\endgroup$
    – Aksakal
    Commented Mar 21, 2018 at 16:30

3 Answers 3

Reset to default
1
$\begingroup$

I found this article to be very helpful in my case:

https://mapr.com/blog/deep-learning-tensorflow/

Using this basic RNN structure, I was able to predict the outcome of the next timestep. By centering all events to the nearest minute, the network was able to recognize the pattern that correlates within the timeline.

Improve this answer
$\endgroup$
0
$\begingroup$

A simple approach would be to treat each event type as independent, and then to build one model per event type. If you expect events to happen on a regular schedule, then an informative feature could be time-since-last-event.

To evaluate the viability of such a model one should do some Exploratory Data Analysis and plot the histograms of such features and analyze whether outliers are present and visible.

If it looks reasonable then one could fit a model to the features. If the features are continuous and normally distributed, the distance from the normalized distribution might be a decent anomaly score. That is easy to compute for a single feature (univariate). For multiple features one would use something like EllipticEnvelope or GaussianMixtureModel.

One could build a multi-event anomaly scoring model using the anomaly scores fro the per-event models.

I see that there are also failure/complete status for events. One might summarize those over a time-period and compute the (time-averaged) failure-rate. Either per-event-type or across all. This one could also build an anomaly detection model on. Perhaps just simple thresholding.

Improve this answer
$\endgroup$
-5
$\begingroup$

There are several ways with which you can tackle this. Before jumping into designing any models standardize your data. Your data seems unlabeled, so initially, what you can do is perform a t-SNE visualization on it which will give you a lot insights to your data. Based on its result you can develop more sensible models which can group the samples into normal ones and anomalies. More on t-SNE here

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ Welcome to stats.SE! The help center has some good information about asking/answering questions. Detailed answers tend to be best. Can you elaborate a bit more? For example, how would t-SNE be used with time series data and discrete 'id' inputs, and how would it be used to help design an anomaly detection system? $\endgroup$
    – user20160
    Commented Mar 6, 2017 at 14:40
  • $\begingroup$ I'm looking into visualising the data with t-SNE per your suggestion, but I'm not sure how far will I get with it. We have several more features to add that I didn't mention, I will edit and add to the post. I still can't figure out what to do once I get visualization working $\endgroup$
    – XOmri
    Commented Mar 6, 2017 at 18:08
  • $\begingroup$ Moreover, t-SNE highly depends on hyperparameters and the does not preserve distances, so how exactly would you find outliers based on it..? Yes it would let you find strange points, but this would be cherry picking. $\endgroup$
    – Tim
    Commented Nov 28, 2018 at 8:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.