Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
1 vote
0 answers
11 views

Find combination of time-dependent values that shows largest anomaly after event

What I have are timeseries of about 500 variables (which are more or less correlated to each other). The variables refer to standardized anomalies (mean=0, std=1). I now define certain event dates in ...
user avatar
  • 11
0 votes
0 answers
11 views

The reason behind fixing the IQR coefficient value in Interquartile Range method

To find the outliers, one common approach is to use the Interquartile Range method, especially when you know your data does not follow Gaussian distribution. ...
user avatar
  • 151
0 votes
1 answer
47 views

Do we need to split the data for Unsupervised Anomaly Detection?

I'm struggling with understanding the concept of splitting data for unsupervised anomaly/outlier detection. You can find all approaches here. I found some papers and implementations that didn't split ...
user avatar
  • 385
0 votes
0 answers
13 views

Multiple Timeseries Anomaly Detection - identifying feature which is anomalous?

I have a multivariate time series where I have features such as: temperature set point energy used relative humidity, etc. Currently, I'm creating univariate anomaly detection models in Python using ...
user avatar
2 votes
0 answers
47 views

Anomaly detection and root cause analysis

ARIMA is widely used for anomaly detection on time-series data e.g. stock price prediction. ARIMA assumes that future value of a variable (stock price in our case) is dependent on its previous values. ...
user avatar
0 votes
0 answers
32 views

One anomaly detection model for all industries

Background - I'm creating a time-series anomaly detection (TSAD) model for the wifi throughput. My customers are 2 banks, 5 retail stores, 4 universities, 6 hospitals. Currently, I have 2 options to ...
user avatar
0 votes
1 answer
24 views

What are the best-practices for validating phone records? [closed]

I have a bunch of telephone data including information on call start time, end time, and duration. I am trying to evaluate the quality of the dataset to determine if the phone call data are legitimate ...
user avatar
  • 494
0 votes
0 answers
31 views

Is good to use binary images for deep learning?

I am trying to make a solar event detector from spectrograms. My problem is, I don't know how to approach the problem properly. I have generated high contrast images from the original data source in ...
user avatar
  • 1
0 votes
0 answers
10 views

What is the appropriate test for comparing detection times?

I am evaluating different strategies for intermittent surveillance of continuous physiologic processes (e.g., heart rate, with a goal of detecting anomalies in timely fashion without being able to ...
user avatar
  • 1
2 votes
3 answers
35 views

Detect dramatic increase in proportion among multiple classes

I would like to use some tool to detect spike in one class's proportion. Assume I received roughly the same percentage of red, blue and yellow candies throughout time. That means the absolute number ...
user avatar
  • 21
0 votes
0 answers
19 views

Anomaly Detection when partitioning totals by groups

I am trying to find a statistical method to identify anomalies in votes. I have a dataset including all the vote for every party in every voting table in every polling station. These votes are counted ...
user avatar
0 votes
0 answers
40 views

Detect periodes of irregular patterns in time series data

This plot shows hourly time series data of a households power usage. The house is only occupied for short periods. What simple alg. or technique can I use to find the start of these irregularities? ...
user avatar
0 votes
0 answers
21 views

Compare three different algorithms for anomaly detection

I have 3 different anomaly detection algorithms, that I tested on a mock dataset of 5 elements. The output of the first and second algorithms, that implement an LSTM, is true/false according to if ...
user avatar
  • 115
0 votes
0 answers
49 views

How to encode categorical data with a lot of unique values and streaming data for anomaly detection

I'm working on a Anomaly Detection problem with streaming data, where i use Robust Random Cut Forest (RRCF). I have 295.000+ rows to start with and there is more data coming in. The problem is when ...
user avatar
0 votes
0 answers
33 views

predict anomaly before it occurs

I have some samples that move on a rail for a few minutes. During this time, some forces act on these samples. For example, I have M samples, for each sample, I have N features that are measured L ...
user avatar
  • 113
0 votes
1 answer
49 views

Is anomaly detection only done using deep learning? Or can you use something like a random forest model to detect anomalies?

Is anomaly detection only done using deep learning? Or can you use something like a random forest model to detect anomalies? For reference, I'm performing classification with a binary response. My ...
user avatar
  • 133
2 votes
0 answers
62 views

Why do I get different results for the Isolation Forest after using PCA?

I am working on an unsupervised anomaly detection project and used the Isolation Forest and AutoEncoders (a normal one and a VAE) to detect anomalies. The AutoEncoders' prediction are identical but ...
user avatar
1 vote
0 answers
40 views

which non parametric test to use for anomalous NN model outputs

Assume I have a bunch of trained NN models for classifying MNIST. All of them except one was trained on the same training set while the one was trianed on a different training set (could have ...
user avatar
  • 223
1 vote
0 answers
29 views

Should I have more trees than dimensions for the Isolation Forest?

I have a dataset which has 200 dimensions after pre-processing. I read multiple times that 100 is the recommended number of trees for the Isolation Forest. Since each tree chooses one feature randomly,...
user avatar
0 votes
0 answers
28 views

Height limits relation to anomaly score in Isolation Forest

I am trying to implement the Isolation Forest algorithm in Python and faced an issue when dealing with the max_depth and the height limit (l) from the white paper. (See the 2. set height limit l) ...
user avatar
1 vote
0 answers
58 views

Standardized Euclidean Distance over variables distributed as $\chi^2$

I sample $n$ dimension vectors (each sample is a vector). My objective is the detection of outliers. In case those elements would distribute normally, for outlier detection, I could use Standardized ...
user avatar
1 vote
0 answers
30 views

How to add account_numbers as input to neural network

Let's say we have a case of money laundering detection, and the only identification for customer and business is their bank_account numbers. How can we encode them for the input to neural networks. ...
user avatar
1 vote
1 answer
43 views

prepare data for autoencoder

I have for example 500 engines and for each engine I have 100 features and each feature is measured 1000 times. It means the data table is: 500,000 rows and 100 columns. (for each engine I have only a ...
user avatar
  • 113
0 votes
0 answers
29 views

LSTM Autoencoder for online anomaly detection

I would like to use an LSTM-Autoencoder for an anomaly detection task, but in an online setting, meaning we are observing the data as it is streaming in. What I would like to do is given some discrete ...
user avatar
0 votes
0 answers
67 views

Anomaly detection with strong prior assumptions about data generating process

I have data that can be described using the model $$ y \sim \mathcal{N}\big(f(x; \Theta), \, \sigma^2) $$ where $f$ is some function with known functional form, but unknown parameters. I also can make ...
user avatar
  • 113k
1 vote
1 answer
33 views

Multivariate anomaly detection with explainability

I have several time serieses, based on aggregation of measurements of different sensors. Each sensor create boolean measurements every hour, and I aggregate them to corresponding time series. For ...
user avatar
  • 11
1 vote
1 answer
89 views

Imbalanced classification vs anomaly detection

Currently I have a dataset with a class distribution of 75:25. While the 1st thought that came to my mind was to use the binary classification using logistic regression. Of course, I can upsample ...
user avatar
  • 1,758
0 votes
1 answer
40 views

Is it possible to use softmax for anomaly detection?

Suppose we have a model for classification. Normally the head of the model is a softmax over all the label/categories. Is it a good idea (that is being used in ...
user avatar
  • 165
0 votes
0 answers
20 views

No variation in responses - irrational behavior?

I have responses from individuals in the form of a Likert-type scale (range 1-7 where 1=strongly disagree and 7=strongly agree). There are 15 statements in 5 domains (3 statements per domain) relating ...
user avatar
  • 206
0 votes
0 answers
40 views

Models/techniques for financial data quality

I'm part of a team tasked with assuring data quality of a large database of financial data (credit cards, collaterals, etc..). We're essentially looking for anomalies, such as single outliers or ...
1 vote
0 answers
33 views

Timeseries Anomaly Detection

I am a programmer not much into math. I have daily data of multiple products (total sales count and amount). Manually we can see the anomaly in the count or amount like if there is a dip or hike in ...
user avatar
0 votes
1 answer
59 views

LOF vs k-NN in data with varying density

When the dataset is comprised of regions of varying densities, which technique is more effective for outlier detection Localised Outlier Factor(LoF) or K-Nearest neighbors (KNN)
user avatar
  • 101
1 vote
1 answer
44 views

How to construct training set for anomaly detection?

I am using a K-Nearest-Neighbor calculation as part of an outlier detection method, and I'm trying to decide how to construct the training dataset on which to base my KNN calculation for subsequent ...
user avatar
  • 337
1 vote
1 answer
42 views

Unsupervised anomaly detection and classification with event (log) data

I am trying to detect anomalies in a large set of user log events, where most users would be considered “good” and a small minority would be considered “bad.” There are hundreds of event types, which ...
user avatar
  • 271
1 vote
0 answers
33 views

When do Efficient Decision Algorithms for 1D Anomaly Detection exist (compared to threshold-tests)?

I'm tasked with investigating whether machine learning algorithms can be used to efficiently identify if a certain type of anomaly is present in the temporal spacing of incoming network packets. ...
user avatar
  • 11
0 votes
0 answers
22 views

PCA: does outlier detection make sense with low linear correlation? [duplicate]

I am experimenting PCA to detect outliers based on the reconstruction error. What I do: I start with a 6 dimensions dataset and reduce it to 5 dimensions. Then, I reconstruct the initial dataset and ...
user avatar
  • 16
1 vote
0 answers
16 views

Outlier Detection in Meta-Analysis Models for Observational Studies of Adverse Drug Outcomes using Distributed Networks

I hope you are in good health. My thesis is on outlier detection in meta-analysis models. I will be using a case study from Canadian Network for Observational Drug Effects Studies (CNODES) to detect ...
user avatar
  • 11
0 votes
0 answers
7 views

Anomaly detection for separate repeat measurement sets

I have been looking for some anomaly detection methods for data that has repeat measurement sets for some specific thing that has happened multiple times on different, but not successive times. I mean ...
user avatar
  • 11
0 votes
0 answers
33 views

A method that uses machine operation data to predict how many days until this machine will fail

Suppose there is a machine that operates every day, and time series data of its operating conditions can be obtained for each day. In this case, I want to use machine learning or etc. to detect when ...
user avatar
  • 155
0 votes
0 answers
13 views

What normalization technique for annomaly detection should I use in deep learning, when the test and real-problem inputs are outside the range?

Which normalization technique should I use in deep learning based anomaly detection. if I expect the value to be outside the range of the normalized value, so min-max for sure is going to be excluded ?...
user avatar
0 votes
1 answer
44 views

Synthetic multivariate time series for anomaly detection

I built an anomaly detection classifier which worked perfectly with the anomaly detection task in my dataset (multivariate time series). Now I'm trying to understand what are its weakness and my idea ...
user avatar
  • 115
0 votes
1 answer
81 views

Heteroskedastic time series outlier analysis using machine learning

Is anyone aware of machine learning models that are able to deal with heteroskedasticity in time series, when trying to detect outliers? There are a lot of anomaly detection tools out there (like k-...
user avatar
1 vote
2 answers
26 views

Periods of High Activity Detection

So I have some sensor data (time series) of heart rate of some users. I want to detect the times they start and finish exercising. The data is sensor readings of heart rate every second, it's ...
user avatar
0 votes
0 answers
74 views

What is the purpose of covariance-matrix in CMGOS?

I'm working on the CMGOS outlier detection algorithm in the Rapidminer tool. Clustering-based Multivariate Gaussian Outlier Score (CMGOS) is a ...
user avatar
  • 2,391
0 votes
1 answer
48 views

Build ROC curve with only labels and predictions

I built different models to approach the anomaly detection problem and I'd like to plot a ROC curve to see how do they perform on my datasets. Both the models are unsupervised neural networks and they ...
user avatar
  • 115
1 vote
0 answers
56 views

train/validate/test split for time series anomaly detection

I'm trying to perform a multivariate time series anomaly detection. I have training data that consists of "normal" data. I train on this data and detect anomalies on the test set that ...
user avatar
0 votes
1 answer
435 views

How to evaluate unsupervised Anomaly Detection using k-means

I'm trying out different anomaly detection models and would love to hear opinion on my idea from somebody experienced. My goal is to perform anomaly detection with different models and to give each ...
user avatar
  • 3
1 vote
1 answer
54 views

In anomaly detection of time series, should global outliers and contextual outliers be separated?

I am trying to create a pipeline in Python which automatically identifies global and contextual anomalies of a time series. Which one of these approaches do you believe is more correct? Method 1) ...
user avatar
  • 151
0 votes
1 answer
63 views

Unevenly spaced time-series forecasting and anomaly detection for an industrial usecase

I am currently working on a PhD project for a car manufacturing company, which basically consists of creating a predictive maintenance application for the machines that are currently used to fill the ...
user avatar
  • 1
0 votes
1 answer
68 views

How to interpret Isolation Forest results on variations of train/test sets?

I have a labelled dataset, originally intended for classification or clustering tasks, whose minority class is at 10%. I am investigating whether this problem can be tackled with anomaly detection ...
user avatar

1
2 3 4 5
9