0
$\begingroup$

I am attempting to find anomalies in accounting data (similar to this study: https://arxiv.org/pdf/1709.05254.pdf). I don't have any labeled data, so this attempt needs to be unsupervised. I am having trouble getting off the ground, because I have no baseline on what normal accounting data (i.e. journal entries) should look like.

I am primarily looking for local anomalies - anomalies that are dependent on the features (i.e. journal entry amount, GL account, number of lines, etc.) occurring in a specific combination that is different from the rest of the data.

I have thought of (and deployed some) clustering methods, but I have no way to know whether the clusters mean anything, because I don't know what normal behavior looks like. How do I determine the baseline of a new dataset without knowing what an anomaly looks like? Can anyone point me in the direction of an anomaly detection method that is unsupervised and works on existing data?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

You can use OneClassSVM or IsolationForest. The purpose of these algorithms are to find what is normal by learning the distribution of the normal things. You need to fit them with normal data, then they will do a binary classification and say what is not normal.

This is also the idea behind auto encoder in anomaly detection (you point out in your article). You expect your anomaly to be rare, so by learning an auto encoder, you should detect the anomaly by getting the data which have bad reconstruction rate.

If you do clustering, you can try a DBSCAN.

Anyway, all those algorithms are not magic. They need parameters, and according to how you set them you will detect different things. For example, in DBSCAN you need to set the distance (or the similarity, to say which data are similar) and some parameters saying your tolerance to noise.

The best way in anomaly detection is to try out those algorithms and little by little get more and more insight on the data. When you will know well your data, you can do good feature extraction which will feed well your algorithm (in fact this is just data science life :D)

Improve this answer
$\endgroup$
3
  • $\begingroup$ this was very helpful. I just stumbled upon sklearns implementation of the outlier detection algorithms, including IsolationForest. When you say "fit them with normal data", is it fine that there may be anomalies in the data? For instance, if I have 10,000 journal entries, and I have no idea which are anomalies, could IsolationForest or LocalOutlierFactor potentially identify the anomalies? (With the proper parameters/knowledge about the data, of course!) LocalOutlierFactor seems to address this by having implementations for novel detection and local detection. $\endgroup$
    – OverflowingTheGlass
    Commented Jul 9, 2019 at 18:57
  • $\begingroup$ Also, which outlier detections methods work well with more than two features? $\endgroup$
    – OverflowingTheGlass
    Commented Jul 9, 2019 at 19:04
  • $\begingroup$ It should be ok as long as you are not with only anomalies. There is a contamination rate for this purpose. They all works with more than two features. I don't know your dataset so it is hard to go further. $\endgroup$
    – PauZen
    Commented Jul 9, 2019 at 19:05
1
$\begingroup$

There is plenty of literature on unsupervised outlier detection.

For example the Local Outlier Factor (Wikipedia), and a simple k-nearest-neighbor-distance based approaches seem to work for many data sets.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.