Skip to main content

Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
2 votes
0 answers
101 views

Anomaly detection for Multivariate Time-Series data from multiple sensors

I work with tabular time-series data from multiple sensors and my goal is to detect abnormal behavior in battery discharge. Here is an example of data (example contains records only for one device ...
mz2300's user avatar
  • 71
0 votes
0 answers
27 views

Very Specific Plateaus in Time Series Data

I am looking at time series data of the depth of water in different pipes. There is a rare occurrence where extreme amounts of water are trying to get into the pipe, but since it is full the water ...
monkey's user avatar
  • 1
2 votes
2 answers
472 views

Methods for Detecting outliers in a time series

I have a question on detecting the outliers in a time series like PPI, CPI, inflation,...etc.) Which method should I use? How can I precisely detect these outliers in a test or a method? Please ...
1 vote
1 answer
30 views

(Multivariate) anomaly detection of (redundant) sensor data

I’m currently working on my master thesis and I’m looking for some inputs for the following situation: I have data of 2-20 sensors all measuring the same variable at 1-3 different locations in 15mins-...
Alexander's user avatar
3 votes
2 answers
74 views

Can an Anomaly Detector be Tested with Data that it Labeled?

Is it wrong to leverage a model to label data, then perform a train/test split to evaluate the performance of said model? Assume I have an unlabeled data set where the missing labels are a binary ...
noNameTed's user avatar
  • 135
0 votes
0 answers
11 views

Convert Multivariate PDF to Anomaly based Risk Score on scale 0 to 1

I have defined a multi-variate normal distribution over a number of risk factors. Currently the PDF over the risk factor observations ranges from infinitesimally values to a maximum of ~2.1. While the ...
noob's user avatar
  • 23
2 votes
1 answer
42 views

Interpreting Mass-Volume as an evaluation criterion for unsupervised anomaly detection

I have found this paper How to Evaluate the Quality of Unsupervised Anomaly Detection Algorithms? by Nicolas Goix that talks about evaluation of unsupervised anomaly scoring functions by the use of ...
deblue's user avatar
  • 243
5 votes
2 answers
496 views

Detect rare high-value measurements in a series of measurements

We do a measurement on 1000 samples to detect if a chemical element A is present, and for each measurement, two cases can happen : the element A is not present, and the values we get are a "...
Basj's user avatar
  • 632
3 votes
1 answer
93 views

Anomaly detection for seasonal data using interquartile range (IQR)

I need to create an alerting bot for anomaly on data regarding funnel monitoring of different product on my company website. The specific metrics I need to monitor are: The conversion rate between ...
Porridge's user avatar
0 votes
0 answers
17 views

Cost function for time series anomaly detection with limited labelled anomalies

Given a time series $y_1, \dots, y_n$, I will fit some models to the data and I want to choose one for anomaly identification. I'm interested in a cost function that rewards a model whose fitted ...
Alex's user avatar
  • 722
0 votes
1 answer
8 views

How to Define Equipment Churn in Laboratory Service Data Without Explicit Churn Labels?

I'm working with a comprehensive dataset spanning 20 years of service records for laboratory equipment owned by various customers. This dataset captures intricate details, such as the equipment ID, ...
tlengman's user avatar
1 vote
1 answer
56 views

Noise Removal for Consistent Anomaly Detection in Multi-Dimensional Time Series Using Matrix Profile

In an online anomaly detection task involving multiple time series, I compute the left matrix profile using non-normalized Euclidean distances for each of the time series (Figure 1). However, since ...
Vlad's user avatar
  • 183
2 votes
1 answer
44 views

Addressing prolonged high matrix profile values in anomaly detection

In an anomaly detection task, I have a data stream where each new data point is generated every 5 minutes. When a new data point arrives, I compute the matrix profile using Stumpy's stumpi function. ...
Vlad's user avatar
  • 183
1 vote
0 answers
29 views

Is using aggregated usage data more efficient than using flattened usage data to build a ML model in anomaly detection? [closed]

We're tracking users' hourly usage on our cloud service and have a risk model that uses aggregated usage data plus other signals to identify potential fraudsters. Basically, it's an anomaly detection ...
Jiayu Zhang's user avatar
1 vote
0 answers
25 views

Outliers in Delta Time column, using data from wireshark

I am currently analyzing data downloaded from Wireshark, focusing on real-time network traffic. I need to perform clusterization on this dataset. However, during the visualization process, I noticed ...
Jaminka's user avatar
  • 11
2 votes
0 answers
23 views

Time Series Anomaly Detection with Class Variables

In ("univariate") time series anomaly detection, what techniques are there to incorporate class variables? For example, in accounting transactional data, we might care about anomalies in the ...
olives's user avatar
  • 31
0 votes
0 answers
68 views

How to validate unsupervised anomaly detection in absence of ground truth?

I am currently working on an unsupervised anomaly detection project and facing a challenge regarding the validation of the model's performance due to the absence of ground truth labels. I am using ...
Camilo Piñón's user avatar
1 vote
0 answers
20 views

Does it make sense to increase the subsampling size for Isolation Forest in the context of sparse matrix data?

I am currently working with a sparse matrix dataset and employing the Isolation Forest algorithm for outlier detection. Given the nature of my data (high dimensionality and sparsity), I am ...
Camilo Piñón's user avatar
0 votes
0 answers
23 views

Similarities between algorithms in anomaly detection in time series

I am interested in detecting outliers (contextual outliers, global outliers) in a 2-dimensional time series data, so I decided to use the tsoutliers library implemented in R, Holt method (I could not ...
Matteo Ceola's user avatar
0 votes
1 answer
51 views

Anomaly detection tsoutliers

I am interested in detecting outliers in a time series data, so I decided to use the tsoutliers library implemented in R. The results that I got from this library were quite disappointing , in fact ...
Matteo Ceola's user avatar
8 votes
1 answer
336 views

How much does an inability to predict an apparent anomaly mean that we lack something in the feature space to distinguish it from business as usual?

I have read a number of questions where the crux is a lamentation that a rare outcome is unable to be predicted by a regression model of some kind. While I understand the desire to be able to reliaby ...
Dave's user avatar
  • 65.1k
1 vote
0 answers
31 views

BEST POSSIBLE WAY to determine significantly high values within zero-inflated univariate continuous distributions

I have more than 50 different distributions, corresponding to 50 different kind of customers, who spend their money in a certain way within a period, being this amount the single variable of interest. ...
0xGolovkin's user avatar
0 votes
0 answers
28 views

Anomaly detection in Multivariate timeseries

I am working on an algorithm which will detect the anomalies in multivariate timeseries. Suppose there is a time series My algorithm will compute two equations: lower_equation_y and upper_equation_y. ...
Raj's user avatar
  • 33
0 votes
1 answer
60 views

Is there a OOD detection method treats OOD samples as another class in multiclass classification?

A simple way I can think of to detect OOD samples is to treat them like another class in a multiclass classification problem. For example, with MNIST, we would modify the network to predict another ...
Lewis Morrow's user avatar
0 votes
1 answer
42 views

Comparison of time series: Cluster behaviors / detect anomalies

I am studying a dataset of time series for different users. The dataset contains records of actions (or registrations) of the users over time. I have data of a whole week for about 80,000 users. ...
tms's user avatar
  • 1
0 votes
0 answers
337 views

Approximating Median Absolute Deviation (MAD) with Rolling Median for Normalization: Trade-offs?

I'm working with a time series dataset and am interested in normalizing the data using the rolling Median Absolute Deviation (MAD). The true MAD is defined as: $$ \text{MAD} = \text{rolling median}(|...
The One's user avatar
  • 225
0 votes
1 answer
87 views

Anomaly Detection in Multivariate and Univariate timeseries

I just started exploring Anomaly detection in timeseries for Univariate, Multivariate timeseries. I read few articles about it, few research papers as well. But every article/research paper has ...
Raj's user avatar
  • 33
0 votes
0 answers
43 views

What is the relationship between bias-variance and sensitivity-specificity for novelty detection?

An over or under-parameterized binary classification model (- vs +) tends to over or under-fit (bias-variance tradeoff). This leads to errors during prediction on unseen data. Depending on if ...
Douw Marx's user avatar
0 votes
0 answers
80 views

Timeseries Anomaly Detection using Rolling Kurtosis?

I'm working on anomaly detection for multiple streaming time series datasets. Due to the vast number of datasets, I'm seeking a scalable, generalized method without resorting to adaptive thresholds ...
The One's user avatar
  • 225
1 vote
1 answer
172 views

batch training with One-Class SVM model

i am working with an anomaly detection problem. i have a large dataset that i cannot fully load it in the memory does batch training works with normal ML models ? does this make sense ???? ...
Mohamed Amine's user avatar
0 votes
0 answers
76 views

Anomaly Detection in Categorical Data

I want to build a system that will detect Anomaly for categorical data. I have a timeseries data like this For metric data these anomalies are calculated Outliers detection Trend Pattern Change I ...
Raj's user avatar
  • 33
0 votes
1 answer
308 views

would One class svm train on only normal data or normal-outlier data both

I was going through this scikit-learn link and I noticed, OneClassSVM is trained on normal and outlier both. Specifically, they are adding the outliers in the following line: ...
sovon's user avatar
  • 101
1 vote
0 answers
24 views

What are the most common fault prediction algorithms?

I have to predict a fault (automotive related) as much in advance as possible. Right now I have found a solution that is somewhat satisfactory (a good number of true positives and a low number of ...
Pigna's user avatar
  • 193
0 votes
0 answers
23 views

Anomaly detection across groups

I'm trying to figure out the best approach for anomaly detection in a particular context. I have data where each observation is a member of a group. I know the groups in the data. Each group has a ...
Michelle's user avatar
4 votes
1 answer
362 views

Comparing frequencies or proportions between two groups to find differences

Suppose I have two groups of users, A and B. In case it is relevant, B is much smaller than A. I have a feature of products purchased across two groups. I want to find items whose popularity is ...
8e9yQBKVlIDwoIVegfkJ's user avatar
1 vote
2 answers
542 views

Difference between anomaly detection and data quality check

I am a beginner in ML and in my company i have been asked to come up with models that can check if there are data quality issue in any given table. It will be an unsupervised learning task and I only ...
Srinath's user avatar
  • 31
0 votes
0 answers
107 views

Convert anomaly scores to probabilities in a one-class way

Im trying to convert anomaly scores into into probabilities. I wanted to do this with logistic regression. The problem is that I can only provide labels for non-anomalous training examples. Training ...
sinpalabras's user avatar
1 vote
1 answer
436 views

Anomaly detection in time-series with categorical data

There are many tutorials/packages in Python to detect anomalies in time-series given that the time-series is numerical. Currently, I have a time-series that is categorical, i.e. the time-series data ...
mommomonthewind's user avatar
1 vote
0 answers
40 views

how to find anomalies for a non-normal distribution with seasonality?

I have a time series broken down by day, and there are gaps in it that I have marked in red: the distribution there is not normal How do we approach modeling a system that will look for anomalies ...
Roman Stasiuk's user avatar
0 votes
1 answer
65 views

How to deal with anomaly detection when the data is produced via multiple resources?

I have a resources lets say $x$ and $y$. These resources produces location data with a timestamp, hence its timeseries data. The data looks like this (resource_uuid,timestamp, location). It might be ...
Joel's user avatar
  • 3
1 vote
1 answer
112 views

Weights Update - Ensemble Models

I must identify if a data point is an outlier or not in a dataset (we don't have labels). I have different unsupervised models to identify the outlier. Then, I normalize the outlier score and I ...
AdrianJoi's user avatar
1 vote
0 answers
45 views

characteristic parameters to judge about residuals of a fit

I am fitting several models to data of unknown size. The models range from linear, quadratic and ODE, however the parameter-identification is always linear and I am using OLS. The parameters of the ...
bluhub's user avatar
  • 11
0 votes
0 answers
48 views

Detecting anomalies in repeating symbolic sequence

I have converted a time series to a symbolic sequence and want to use machine learning to detect anomalies. In this specific case, every time series represents the same process. So a similar, but not ...
sinpalabras's user avatar
0 votes
1 answer
132 views

What is a suitable technique for detecting anomalies in time series data?

I have a problem, where I try to identify if a machine performs an activity when it is not supposed to, or performs it an unusual number of times. I am attempting to this using an anomaly detection ...
Nht_e0's user avatar
  • 33
1 vote
0 answers
52 views

If a time series, which is a sum of some component time series, is spiking, how do I attribute the spike to the components

Let $X_t$ be a time series with $X_t = Y_t + Z_t$. At time $T$, $X_T$ "spikes"; that is to say, $X_{T-1}-X_T$ is over some critical threshold. I would like to assign some "contribution&...
crf's user avatar
  • 309
0 votes
0 answers
29 views

Leads to detect rare events in human related multi-variable time series

I am currently working on a subject and struggling to find a way to tackle it. Ok, so, let's say I want to predict whenever a human is going to faint in a rollercoaster. I have multi-variables time ...
Jules Civel's user avatar
3 votes
2 answers
417 views

Do I want to overfit, when doing outlier detection based on regression?

Imagine, we have speed data of car and we would like to detect, if car speeds up or down more than it should. Do I want to just overfit my model, so the outlier (higher or lower speed) would lead me ...
Mr. Panda's user avatar
  • 313
1 vote
1 answer
264 views

Feature engineering for anomaly detection when feature is directional

I want to train an anomaly detection model for intrusion and fraud detection. I have several features I know are correlated with sketchy behavior. However, those features are "directional" ...
fenmap's user avatar
  • 11
0 votes
1 answer
28 views

In what case is PR AUC higher than ROC AUC?

I am working on an anomaly detection problem and have come across a paper(https://www.ijcai.org/proceedings/2019/0378.pdf), which shows results where in the ROC AUC value for a dataset is 0.566 and ...
Vishal Reddy's user avatar
0 votes
0 answers
104 views

Detect periods of gradual decreases in time-series data

I have some time-series data sets in which, in principle, two types of event are possible: the signal can instantaneously jump up or down; or there can be a gradual decrease in the signal. I want to ...
taitlab's user avatar

1
2 3 4 5
10