1
$\begingroup$

Assuming there is a time series dataset of 3 columns: userid, timestamp, transaction_amount. There are millions of users. Years of data. What might be an easy and fast algo/ models to detect anomalies here?

STL seems to be able to handle 2 columns/ dimension well (for example, timestamp, transaction_amount per day), but doesn't seem to be able to handle 3 columns (userid, timestamp, transaction_amount) here together. Unless my understanding of STL is incorrect.

An unsupervised machine learning algo, Local Outlier Probabilities, seems to be a good fit here and provides good visualization. But when the 3 dimensional dataset is huge like this, the performance seems to be a real issue.

https://github.com/vc1492a/PyNomaly

Kriegel H., Kröger P., Schubert E., Zimek A. LoOP: Local Outlier Probabilities. 18th ACM conference on Information and knowledge management, CIKM (2009)

Wonder if anyone might have any thoughts? Thank you!

$\endgroup$
0
$\begingroup$

userid, timestamp are not right variables to pass for local outlier probability as these are categorical. Where do u get distance for user ids?

1) First define what is abnormality for you? 'I am going to find abnormality' is incorrect problem statement. Have a clear definition of abnormality. is it just outlier so you can think of LOF or other algo.

2) sometimes just a distribution of data would help in detecting outliers.

3) you can also thing of other unsupervised algos like-

angle based outlier detection- https://machinelearningstories.blogspot.com/2018/08/anomaly-detection-in-high-dimensional.html

connectivity based outlier detection- https://machinelearningstories.blogspot.com/2018/09/connectivity-based-outlier-detection.html

going back to first point -identify abnoramlity definition first. I see only 1 variable in your data -amount. Do you even need any algo? :P

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.