Unsupervised Anomaly Detection with groups

Question

Let's say we are a bank and are interested in catching fraudulent customers. We gather ~100.000 independent samples of 40 independent variables and 4 are behavioral variables (what a customer does).

Examples of independent variables (mix of numeric and categorical):

• age
• City where they live
• Social Economic Status, etc.

Then we have four behavioral variables (mix of numeric and categorical):

• Number of transactions
• Number of international transactions
• Sum of money transferred
• Type of product they used that month

Now we're interested in finding the outliers in this set. There is no information on what was a fraudulent customer in the past, so its an unsupervised problem. Given a customers features, is his behavior expected or unexpected?

If we clusters clients based on their independent variables, we would have a featurespace where clients are put together that look alike. Most would then also lie relatively close in the behavior space, but some might have very different behavior. These are the clients we're interested in.

Problems I'm facing:

1. If the behavioral featurespace would be very small (e.g. 4 buckets), I'd use a clustering algorithm for each of these 4 behaviours. However, the featurespace is much bigger and there is no obvious way to reduce this. Combinations of all values in here lead to ~50K.
2. There is very little business domain knowledge on what to expected (e.g. number of outliers). So quite hard to check.
3. I'm unsure how to combine the distance in the featurespace to the distance in the behaviorspace. Are there any algorithms/papers/articles that deal with this?
4. I Haven't really come across people that have seen similar challenges. Is there a good term to describe this problem and google it?

Note: I've used 'groups' in the title to refer to groups of clients that would probably show the same behavior, but its not super indicative of the problem I'm trying to solve. Any suggestions are welcome!

Answer 1

After a lot of searching, I found out this type of problem is actually referred to as conditional outlier detection. So for any others that deal with a similar problem, search for this term and many relevant papers will pop up.

Answer 2

One method you could try is Isolation Forsts. The method works by randomly selecting variables, then randomly selecting a cut-off point for the selected variable and doing this until observations are all “isolated”. This can be repeated to het an ensemble of trees. The easier to separate an observation the more likely (according to this method) the observation is to be an outlier. The resulting trees can be used to give each observation an anomaly score, with values close to one more likely to be anomalies.

Below is text copied from https://en.m.wikipedia.org/wiki/Isolation_forest.

The authors took advantage of two quantitative properties of anomalous data points in a sample:

Few - they are the minority consisting of fewer instances and Different - they have attribute-values that are very different from those of normal instances Since anomalies are "few and different", they are easier to “isolate” compared to normal points. Isolation Forest builds an ensemble of “Isolation Trees” (iTrees) for the data set, and anomalies are the points that have shorter average path lengths on the iTrees.

One thing to consider is the curse of dimensionality applies here, again from the Wikipedia article:

One of the main limitation to standard, distance-based methods is their inefficiency in dealing with high dimensional datasets: The main reason for that is, in a high dimensional space every point is equally sparse, so using a distance-based measure of separation is pretty ineffective. Unfortunately, high-dimensional data also affects the detection performance of iForest, but the performance can be vastly improved by adding a features selection test like Kurtosis to reduce the dimensionality of the sample space.

Answer 3

There's plenty around if you have access to journals; a lot get put up on researchgate or arxiv anyway. Try and find some literature reviews, books or other work which talk about various strategies if you can.

Vector embedding techniques are quite common and might be worth checking out. They can also be extended to sequence data, which you might find useful.

Techniques from graph theory have the advantage of being fairly intuitive to interpret.

GANs have been coming into play more recently.

You might also want to look into community detection and social network analysis.