Questions tagged [anomaly-detection]
Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. This is also known as outlier detection.
468
questions
0
votes
0
answers
9
views
Is there a OOD detection method treats OOD samples as another class in multiclass classification?
A simple way I can think of to detect OOD samples is to treat them like another class in a multiclass classification problem. For example, with MNIST, we would modify the network to predict another ...
0
votes
1
answer
21
views
Comparison of time series: Cluster behaviors / detect anomalies
I am studying a dataset of time series for different users. The dataset contains records of actions (or registrations) of the users over time. I have data of a whole week for about 80,000 users.
...
- 1
0
votes
0
answers
21
views
Approximating Median Absolute Deviation (MAD) with Rolling Median for Normalization: Trade-offs?
I'm working with a time series dataset and am interested in normalizing the data using the rolling Median Absolute Deviation (MAD). The true MAD is defined as:
$$ \text{MAD} = \text{rolling median}(|...
- 285
0
votes
1
answer
33
views
Anomaly Detection in Multivariate and Univariate timeseries
I just started exploring Anomaly detection in timeseries for Univariate, Multivariate timeseries.
I read few articles about it, few research papers as well. But every article/research paper has ...
- 19
0
votes
0
answers
10
views
What is the relationship between bias-variance and sensitivity-specificity for novelty detection?
An over or under-parameterized binary classification model (- vs +) tends to over or under-fit (bias-variance tradeoff). This leads to errors during prediction on unseen data. Depending on if ...
- 53
0
votes
0
answers
21
views
Timeseries Anomaly Detection using Rolling Kurtosis?
I'm working on anomaly detection for multiple streaming time series datasets. Due to the vast number of datasets, I'm seeking a scalable, generalized method without resorting to adaptive thresholds ...
- 285
1
vote
1
answer
26
views
batch training with One-Class SVM model
i am working with an anomaly detection problem.
i have a large dataset that i cannot fully load it in the memory does batch training works with normal ML models ?
does this make sense ????
...
- 171
0
votes
0
answers
38
views
Anomaly Detection in Categorical Data
I want to build a system that will detect Anomaly for categorical data. I have a timeseries data like this
For metric data these anomalies are calculated
Outliers detection
Trend
Pattern
Change
I ...
- 19
0
votes
1
answer
54
views
would One class svm train on only normal data or normal-outlier data both
I was going through this scikit-learn link and I noticed, OneClassSVM is trained on normal and outlier both. Specifically, they are adding the outliers in the following line:
...
- 101
1
vote
0
answers
16
views
What are the most common fault prediction algorithms?
I have to predict a fault (automotive related) as much in advance as possible.
Right now I have found a solution that is somewhat satisfactory (a good number of true positives and a low number of ...
- 173
0
votes
0
answers
16
views
Anomaly detection across groups
I'm trying to figure out the best approach for anomaly detection in a particular context. I have data where each observation is a member of a group. I know the groups in the data. Each group has a ...
- 1
4
votes
1
answer
289
views
Comparing frequencies or proportions between two groups to find differences
Suppose I have two groups of users, A and B. In case it is relevant, B is much smaller than A.
I have a feature of products purchased across two groups. I want to find items whose popularity is ...
0
votes
0
answers
9
views
Figure comparing path length and density (figure 3a) in paper "Isolation-based Anomaly Detection" by Liu, Ting and Zhou
In the paper Isolation-based Anomaly Detection by Liu, Ting and Zhou, figure 3a compares the density and path length for a cluster of anomalies. Figure 3a along with the sub-text has been provided as ...
0
votes
0
answers
9
views
Significance of challenges cited, in creating anomaly scores out of path length, in original paper for Isolation Forest
In the original paper on Isolation Forest by Liu, Ting and Zhou, the authors cite some problems in creating anomaly scores out of the path length $h(x)$ of a point $x$ in an iTree. The text reads as ...
0
votes
0
answers
53
views
Data Quality check in time series data
I have time series data. The table contains date, no of orders, no of products sold, no of returns, no of walk in customers, etc. I want to create a model which takes 12 months of data as input and ...
- 31
1
vote
2
answers
191
views
Difference between anomaly detection and data quality check
I am a beginner in ML and in my company i have been asked to come up with models that can check if there are data quality issue in any given table. It will be an unsupervised learning task and I only ...
- 31
0
votes
0
answers
17
views
Anomaly Detection for Categorical Data
Lets say i have a table which has 10 categorical features for each customer and these features are recorded on a daily grain. This means that I have 10 categorical time series for each customer.Now I ...
- 31
0
votes
0
answers
26
views
Anomaly Detection for Log Counts
I have a dataset of Log count data per day (for DNS log) for over a year. I need to create an anomaly detection method to predict future anomalous counts to detect unusual log count. Initially, I ...
0
votes
0
answers
55
views
Convert anomaly scores to probabilities in a one-class way
Im trying to convert anomaly scores into into probabilities. I wanted to do this with logistic regression. The problem is that I can only provide labels for non-anomalous training examples. Training ...
- 21
0
votes
1
answer
158
views
Anomaly detection in time-series with categorical data
There are many tutorials/packages in Python to detect anomalies in time-series given that the time-series is numerical.
Currently, I have a time-series that is categorical, i.e. the time-series data ...
- 947
1
vote
0
answers
25
views
how to find anomalies for a non-normal distribution with seasonality?
I have a time series broken down by day, and there are gaps in it that I have marked in red:
the distribution there is not normal
How do we approach modeling a system that will look for anomalies ...
0
votes
0
answers
39
views
How to get the dynamic threshold to detect rare categories from the univariate categorical variable?
I was working on combining the "rare" categories into a single category/group "others" automatically in a univariate categorical variable.
Suppose I have A, B, C, D, and E ...
- 123
0
votes
1
answer
48
views
How to deal with anomaly detection when the data is produced via multiple resources?
I have a resources lets say $x$ and $y$. These resources produces location data with a timestamp, hence its timeseries data. The data looks like this (resource_uuid,timestamp, location).
It might be ...
- 3
1
vote
1
answer
76
views
Weights Update - Ensemble Models
I must identify if a data point is an outlier or not in a dataset (we don't have labels). I have different unsupervised models to identify the outlier. Then, I normalize the outlier score and I ...
- 11
1
vote
0
answers
43
views
characteristic parameters to judge about residuals of a fit
I am fitting several models to data of unknown size. The models range from linear, quadratic and ODE, however the parameter-identification is always linear and I am using OLS. The parameters of the ...
- 11
0
votes
0
answers
44
views
Detecting anomalies in repeating symbolic sequence
I have converted a time series to a symbolic sequence and want to use machine learning to detect anomalies. In this specific case, every time series represents the same process. So a similar, but not ...
- 21
0
votes
1
answer
91
views
What is a suitable technique for detecting anomalies in time series data?
I have a problem, where I try to identify if a machine performs an activity when it is not supposed to, or performs it an unusual number of times.
I am attempting to this using an anomaly detection ...
- 33
1
vote
0
answers
52
views
If a time series, which is a sum of some component time series, is spiking, how do I attribute the spike to the components
Let $X_t$ be a time series with $X_t = Y_t + Z_t$. At time $T$, $X_T$ "spikes"; that is to say, $X_{T-1}-X_T$ is over some critical threshold. I would like to assign some "contribution&...
- 299
0
votes
0
answers
28
views
Leads to detect rare events in human related multi-variable time series
I am currently working on a subject and struggling to find a way to tackle it.
Ok, so, let's say I want to predict whenever a human is going to faint in a rollercoaster. I have multi-variables time ...
- 101
3
votes
2
answers
334
views
Do I want to overfit, when doing outlier detection based on regression?
Imagine, we have speed data of car and we would like to detect, if car speeds up or down more than it should.
Do I want to just overfit my model, so the outlier (higher or lower speed) would lead me ...
- 263
0
votes
1
answer
155
views
Feature engineering for anomaly detection when feature is directional
I want to train an anomaly detection model for intrusion and fraud detection. I have several features I know are correlated with sketchy behavior. However, those features are "directional" ...
- 1
0
votes
1
answer
21
views
In what case is PR AUC higher than ROC AUC?
I am working on an anomaly detection problem and have come across a paper(https://www.ijcai.org/proceedings/2019/0378.pdf), which shows results where in the ROC AUC value for a dataset is 0.566 and ...
0
votes
0
answers
20
views
Performance metrics for time series anomaly detection with rolling horizon
I have a time series problem, where I am trying to find indication of break-down of machines in a factory based on a set of features related to said machines and their function (e.g. hours working, ...
- 1
0
votes
0
answers
72
views
Detect periods of gradual decreases in time-series data
I have some time-series data sets in which, in principle, two types of event are possible: the signal can instantaneously jump up or down; or there can be a gradual decrease in the signal. I want to ...
- 1
0
votes
0
answers
29
views
How to classify unseen data as anomaly
I trained a CNN model with 6 different classes (labels are 0-5) and I am getting more than 90% accuracy out of it. It can correctly classify the classed. I am actually trying to detect anomaly with it....
- 11
1
vote
1
answer
136
views
For time-series data what is the difference between decomposing and detrending data?
To use AR/MA/ARMA models, the data needs to be stationary. Detrending data by taking differences between sequential data points seems to do this. However, does decomposing the data do the same thing? ...
- 111
1
vote
1
answer
63
views
Autoencoders for anomaly detection
Autoencoders cannot process large-scale data sets efficiently, due to its serial implementation. can you please explain why it happens?
Thanks in advance!!
1
vote
1
answer
55
views
In which normal distribution you are more likely to find anomalies?
(Sorry if talk in layman terms)
Consider this you have 2 groups of populations. Let's assume we measure intelligence here. Let's assume the distributions are perfectly normal.
One group is centered ...
- 111
1
vote
1
answer
56
views
How to handle machinery shut-down in Time Series for Anomaly Detection
I have this data coming from sensors installed in a industrial machinery and my ultimate goal is implementing an anomaly detection method on it.
Now, the data is quite noisy and with lots of missing ...
- 23
0
votes
0
answers
53
views
Are ratios and percentages univariate or multivariate?
Edited
I'm working on an anomaly detection project where I have a set of independent variables and dependent variables. As I have ratio and percentage data, I want to know how to document ratios and ...
0
votes
0
answers
32
views
Should you use mean difference between measurements or min-max difference to detect outliers?
I have a dataset which has temperature measurements for every minute in a certain time period.
I want to focus on 10 minute intervals and determine whether two adjacent 10 minute intervals differ ...
- 47
1
vote
1
answer
69
views
OneClassSVM performs better when trained on pure data (inliers only)
I have a dataset which has some outliers and majority of the class is inliers (not anomalous). I am trying to train a OneClassSVM/...
- 113
3
votes
0
answers
73
views
loss function for supervised anomaly detection in time series [closed]
I have a supervised anomaly detection problem in a time series data, which the dataset has three columns:
datetime
value(a float number)
label(1 for anomaly, 0 for normal)
It's common that the ...
1
vote
0
answers
216
views
How to detect low and high flow outliers with seasonal time series data in R?
I have a dataset recording daily river flow from 1976 to 2017. I want to find out unusually high (potential flood) or low (potential drought) flow values from that datatset. What's the best way to ...
- 111
0
votes
0
answers
38
views
Formula of composite daily anomaly
If a daily anomaly is a difference between the daily mean from the corresponding daily climatology for the sought calendar day of the year, how to calculate the composite daily anomalies? Is there a ...
- 23
0
votes
1
answer
21
views
How to perform anomaly classification in multivariate time series when anomalies appear in different time windows?
I'm dealing with anomaly classification in multivariate time series.
However, in my problem, the different classes of anomalies can be detected when considering different time windows. That is, some ...
- 678
2
votes
0
answers
245
views
How to normalize count data with possible large values and lots of zeros? [closed]
I am working on an anomaly detection problem in network data.
My dataset consists of aggregated data over a fixed period of time (1mn window). For numerical features we store averages and for ...
- 21
1
vote
1
answer
116
views
how to detect anomalies in data
I want to get ideas of how to approach this problem and what is the different effect of each one.
Assume you have a set of time series (several variables measured always at the same time) and you ...
- 11
2
votes
1
answer
86
views
Combining time series anomaly measures
suppose I have different "scores" that can be computed for every data point in a time series. The score quantifies how anomalous the point is. The scores are very different in nature, so its ...
- 21
6
votes
1
answer
736
views
Outlier/anomaly detection on histograms
So, the idea is that I have many histograms, each one representing results for something. So, I have histogram_1 for object_1, histogram_2 for object_2,...,histogram_20 for object_20. How can throw ...
- 137