Questions tagged [anomaly-detection]
Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. This is also known as outlier detection.
43
questions
4
votes
2
answers
2k
views
Strange encoding for categorical features
I am reading through https://arxiv.org/pdf/1609.06676.pdf which presents an extension of the isolation forest algorithm so that categorical features may be taken into account. On page 5, the authors ...
- 707
7
votes
1
answer
295
views
How much does an inability to predict an apparent anomaly mean that we lack something in the feature space to distinguish it from business as usual?
I have read a number of questions where the crux is a lamentation that a rare outcome is unable to be predicted by a regression model of some kind. While I understand the desire to be able to reliaby ...
- 62.5k
81
votes
9
answers
97k
views
What algorithm should I use to detect anomalies on time-series?
Background
I'm working in Network Operations Center, we monitor computer systems and their performance. One of the key metrics to monitor is a number of visitors\customers currently connected to our ...
- 911
26
votes
5
answers
28k
views
Algorithms for Time Series Anomaly Detection
I'm currently using Twitter's AnomalyDetection in R: https://github.com/twitter/AnomalyDetection. This algorithm provides time series anomaly detection for data with seasonality.
Question: are there ...
- 471
21
votes
4
answers
32k
views
Encoding of categorical variables with high cardinality
For unsupervised anomaly detection / fraud analytics on credit card data (where I don't have labeled fraudulent cases), there are a lot of variables to consider. The data is of mixed type with ...
- 707
1
vote
1
answer
2k
views
Estimating When A Time Series with Random Spikes Crosses a Threshold for the First Time
tl;dr Is there a way to estimate when a random spike in a time series would cross a threshold for the first time?
The following is data of my performance in the game Super Hexagon, whose goal is to ...
- 143
0
votes
2
answers
561
views
How to create a training set when I have no clue about my data?
I have a dataset consisting of numbers representing the values of a KPI (Key performance indicator) collected over a period of time. I would like to implement an algorithm to classify the data as ...
- 121
19
votes
1
answer
2k
views
Robust PCA vs. robust Mahalanobis distance for outlier detection
Robust PCA (as developed by Candes et al 2009 or better yet Netrepalli et al 2014) is a popular method for multivariate outlier detection, but Mahalanobis distance can also be used for outlier ...
- 1,322
12
votes
1
answer
2k
views
Are time series motifs and the Matrix profile algorithm a good fit for my problem?
I have huge multivariate time series to analyze (Terabytes of data) and I need fast, scalable algorithms for mainly two tasks:
finding similar patterns among time series. For example, imagine I ...
- 18k
11
votes
3
answers
17k
views
Time Series Anomaly Detection with Python
I need to implement anomaly detection on several time-series datasets. I've never done this before and was hoping for some advice. I'm very comfortable with python, so I would prefer the solution be ...
- 471
7
votes
1
answer
915
views
Outlier/anomaly detection on histograms
So, the idea is that I have many histograms, each one representing results for something. So, I have histogram_1 for object_1, histogram_2 for object_2,...,histogram_20 for object_20. How can throw ...
- 157
5
votes
2
answers
10k
views
How to perform Validation on Unsupervised learning?
Since I consider Unsupervised learning, I don't have any ground truth to compare with, during the validation phase. So, is there any standard method to deal with it?
Additional informations:
in my ...
- 61
2
votes
2
answers
950
views
How to choose a method for binary classifier based on only positive and unlabelled examples?
I need to build a binary classifier with machine learning, as I fail to manually choose a combination of features to achieve minimal fraction of false positives.
What is best practice for choosing a ...
- 139
1
vote
1
answer
1k
views
Do we need to split the data for Unsupervised Anomaly Detection?
I'm struggling with understanding the concept of splitting data for unsupervised anomaly/outlier detection. You can find all approaches here. I found some papers and implementations that didn't split ...
- 421
22
votes
7
answers
29k
views
Difference between Anomaly and Outlier
What is the difference between Outlier and Anomaly in the context of machine learning. My understanding is that both of them refer to the same thing.
- 321
21
votes
2
answers
10k
views
Anomaly Detection with Dummy Features (and other Discrete/Categorical Features)
tl;dr
What is the recommended way to deal with discrete data when performing anomaly detection?
What is the recommended way to deal with ...
- 313
13
votes
1
answer
14k
views
Difference between Outlier and Inlier
I have stumbled upon the term inlier in the LOF measure (Local Outlier Factor), I am familiar with the term of outliers (well basically liers - instances which doesn't behave as the rest of the ...
- 369
10
votes
1
answer
3k
views
Anomaly detection using PCA reconstruction error
I would like to use PCA as a method of anomaly detection, however I'm wondering how this is done exactly (I'm using prcomp in R).
I'm really questioning the ...
- 835
9
votes
4
answers
8k
views
Metrics for one-class classification
How do you calculate precision and recall in one class classification?
In other words in one class classification, we just have TP(True Positive) and FN(False Negative). Which metrics we should use ...
- 91
9
votes
1
answer
18k
views
How to use PCA to detect outliers?
A PCA will reduce the dimensionality of the original data and construct a subspace generated by eigenvectors of which each represents the (next) highest variance to explain the data.
Let's start at ...
- 3,443
9
votes
3
answers
5k
views
Does a statistically significant correlation always give predictive power?
Suppose you're trying to predict anomalies. That is, consider the case where you have a data set that has a column called result. Suppose the data set has 365 rows and result has a value of 1 in ...
- 827
8
votes
2
answers
10k
views
Difference between contextual anomaly and collective anomaly
Contextual and collective anomalies are defined as follows (source):
Contextual Anomalies. If a data instance is anomalous in a specific context
(but not otherwise), then it is termed as a contextual ...
- 203
8
votes
2
answers
6k
views
Time series anomaly detection
I am tasked to develop an anomaly detection system for data organised in many 1D (can be more than 1D if I choose, but I think that will complicate the problem even more) daily time series. The series ...
- 81
7
votes
1
answer
7k
views
How to estimate the scale factor for MAD for a non-normal distribution?
I understand that the scale factor for normally distributed data is 1.4826 to convert it to a pseudo standard deviation like quantity which could be used with the median for determining confidence ...
- 333
7
votes
1
answer
8k
views
Anomaly detection with gaussian mixture models
I am new to the topic, and I am trying to understand how it is possible to perform anomaly detection by using gaussian mixture models.
Could you please give me some hints about literature on the topic?...
- 71
6
votes
1
answer
6k
views
Intuitive explanation of Minimum Covariance Determinant (MCD)
I am an undergrad working on Anomaly Detection on an 8 dimensional dataset, with PYOD, which relies on the MCD in the sklearn's MinCovDet. I tried reading Minimum Covariance Determinant and Extensions,...
- 163
6
votes
1
answer
484
views
Any good reference books/material to help me build a txn level fraud detection model?
I am looking for a book/case study etc on how to build a fraud detection model at the transaction level. Something applied rather than theoretical would be really helpful.
user3081
6
votes
2
answers
9k
views
How to get top features that contribute to anomalies in Isolation forest
I am using Isolation forest for anomaly detection on multidimensional data. The algorithm is detecting anomalous records with good accuracy. Apart from detecting anomalous records I also need to find ...
- 61
5
votes
0
answers
2k
views
How to normalize input data for autoencoders - anomaly detection
I'm building an autoencoder to identify anomalies on numerical data. The input features have different scales (i.e. some take values from 0 to 5, while others can be much much higher) and most of them ...
- 558
5
votes
1
answer
5k
views
Is Anomaly Detection Supervised or Un-supervised?
AFAIK - One way to process data faster and more efficiently is to detect abnormal events, changes, or shifts in datasets. Anomaly detection, also known as outlier detection is the process of ...
- 2,915
4
votes
1
answer
4k
views
Anomaly detection in multivariate time series data
I am trying to solve an anomaly detection problem that consists of three variables captured over a span of five years. It is an unsupervised problem, and I believe density-based clustering methods ...
- 123
3
votes
1
answer
181
views
Recommendations for anomaly detection
I have a binary classifier. The classifier is trained on both numeric and categorical variables. In a given month, I will have new data coming in, comparable to 5% of the observation count of the ...
- 453
2
votes
2
answers
2k
views
Transfer Learning on Autoencoders?
I want to use the encoder of my autoencoder for feature extraction in an image anomaly detection framework.
For that reason, I thought that pretraining the autoencoder on a large dataset and then fine-...
- 93
2
votes
1
answer
1k
views
Does it make sense to use attention mechanism for seq-2-seq autoencoder for anomaly detection?
So I want to train LSTM sequence to sequence model, autoencoder, for anomaly detection. The idea is to train it on normal samples and when anomaly comes into model it will not be able to reconstruct ...
- 753
2
votes
1
answer
63
views
Using unsupervised learning anomoly detection to detect fraud?
How can I ensure that the detected class will correspond to fraud rather than another outcome, given that this is an unsupervised learning approach? To my understanding, such algorithms (e.g., ...
- 21
2
votes
1
answer
5k
views
Clustering based anomaly detection
I'm trying to implement anomaly detection based on clustering. I'm hopping for confirmation of my approach, and I'm exposing my idea, being aware that I could have miss something in my analysis, so ...
- 2,857
2
votes
0
answers
732
views
NMDS anomaly - data does not support point placement
My data:
Tracking forest communities (via species abundances) in various forest plots across time.
My approach: Non-metric Multidimensional Scaling ordination
I performed NMDS (using ...
- 2,325
2
votes
1
answer
971
views
Use case for Anomaly Detection using the multivariate Gaussian distribution
We have 5000 vehicles of different classes (trucks, small cars, large cars) with 100 sensors in each car measuring fuel consumption, distance traveled, average speed etc for some time period $t$ that ...
- 161
2
votes
1
answer
556
views
How to measure the consistency of improvement on different conditions?
I want to measure whether the speed improvement of method 1 over method 2 is consistent on different conditions. Below are two examples of the speedup values of method 1 over method 2 on 5 conditions.
...
- 119
1
vote
1
answer
349
views
Anomaly detection in time-series with categorical data
There are many tutorials/packages in Python to detect anomalies in time-series given that the time-series is numerical.
Currently, I have a time-series that is categorical, i.e. the time-series data ...
- 987
1
vote
1
answer
3k
views
Anomaly/Outlier detection based on Windows event security logs (logons) using Machine Learning(in Python) [closed]
I am trying to solve the problem of finding anomalies/outliers using event security logs of an individual system. Please find the details below:
Problem Statement: Find anomalies/outliers using event ...
1
vote
1
answer
315
views
Recommendations for textbooks covering current data mining/machine learning techniques for fraud detection?
I work in the health insurance field, but a general treatment of fraud detection methodologies would still be helpful.
So far I've discovered brief articles outlining particular techniques used in ...
- 6,104
0
votes
2
answers
1k
views
Outliers Detection with unlabeled data? [closed]
I have a dataframe with numeric and categorical variables and no target variables and I need to check for multivariate outliers.
Could you suggest a model (using Python) that works good for outliers ...
- 13