Unsupervised anomaly detection and classification with event (log) data I am trying to detect anomalies in a large set of user log events, where most users would be considered “good” and a small minority would be considered “bad.” There are hundreds of event types, which have time stamps and happen sequentially (as most events do). Some of these events are triggered frequently and some are rarely triggered. E.g. the user creates an account, adds a credit card, makes a purchase, changes their profile, adds a picture, etc.
I do have some labels, but they are low quality (probably high precision but low recall), and I would like to be able to detect anomalies not currently in the data. So I’m trying do detect user behavior patterns that are outside the “norm” (which will include most users) in an unsupervised way. To be clear, I want to classify the user, not individual events per se, but want to update my opinion of the user frequently.
Preliminary analysis suggests that signals that may pop will be something like “an unusually high number of event A given this age of account” or “an unusually high number of event A given the number it times event B has happened so far.” As in, “it’s weird that someone would start making so many connection requests before setting up their profile.” You get the idea. We’ve engineered some features, but it’s been slow going, and I’d like to feature engineer at scale.
So my plan as a first pass is, for each event (let’s call it Event A), look at the number of times that Event A has happened so far (cumulative count) and calculate the probability (the percent of existing users, really) that would have at least that many, given the number of times another event has happened (Events B-Z). Event B could be seconds since account creation or number of times a credit card as been added, etc.
I’d just save those probabilities in an array, and repeat for each type of event, creating a 2d array of probabilities. When an event is triggered, that row and column would be updated to reflect the new probabilities.
Then I’d take these 2d arrays, treat them basically as images, and plug them into a clustering algo. The hope is that that most “good” users have relatively similar patterns of behavior (or a few different types) but “bad” users cluster into separate types based on anomalies across a number of dimensions.
My reason for doing this rather than just plugging directly into a NN is for purposes of explainability and monitoring. The ability to update a bit of the data based on new events and rescore without rewriting everything would also be nice, since I want to detect anomalies continuously.
So:
Is this dumb for any reasons I should know?
Is this already a thing that already exists but I don’t know the name of?
 A: I am not sure about what the 2d arrays, that you are referring to, have as row and column labels, maybe event type versus user? Anyway, The standard approach (which might be what you are describing, I am not sure...) is to create for each user a set of m reasonable summary statistics and use them to assign to each user a point in an m-dimensional space. So your users become an m-dimensional point cloud, which you can update as you wish. Next, you look for isolated points or isolated small groups in this point cloud, which would be your outliers/anomalies. Some clustering algorithms can be used here, but methods like isolation forest, OCSVM, kNN, ... (see here for examples and implementations) are often more appropriate.
Of course, decisive is the choice of summary statistics, which is a job for a domain expert like you. And you have already mentioned some like several count statistics per time interval.
Furthermore, it also makes sense to use the time series character of your data, and to do time series based anomaly detection (AD). The idea here is to either compute probabilities over subsequences and to flag those that have very low probability, or to compute probabilities for the next event (prediction) and raise alarm if the actual next event is very unlikely w.r.t. those probabilities. For this time series approach, deep neural networks are performing sometimes quite well, but there are also many classical models you could use, from exponentially smoothing to hidden Markov models or other dynamic Bayesian networks.
