Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
1
vote
0answers
31 views

How to detect anomalies in multiple different IP addresses?

Given that my input data consists of various destination IP addresses and its incoming connections from source IP addresses with country codes during certain timestamps, I would like to detect ...
2
votes
1answer
25 views

Matrix Profile and mean-shift detection

I'm currently working on anomaly detection on time series and one of the discords I'm trying to detect are 'mean-shifts,' i.e. the signal suddenly shifting by a certain value while retaining its ...
0
votes
0answers
14 views

True Positive Rate (TPR), and False Positive Rate (FPR) to evaluate binary classifier in the anomaly detection domain

I am confused with the the binary classifier evaluation criteria in anomaly detection domain for highly imbalanced data. I see that some authors claimed that their model performed excellent to predict ...
0
votes
0answers
15 views

One-class SVM formula

This is the same question on https://datascience.stackexchange.com/questions/96555/one-class-svm-formula. Recently I have been studying one-class SVM and am a little bit confused about the offset $\...
0
votes
0answers
2 views

Minimum distance between pair of points in DBSCAN

I have just used DBSCAN to detect the outliers. If $\epsilon$ is the radius as the given parameter, then a point is outlier means it's distance with all points greater than $\epsilon$, But I have just ...
0
votes
0answers
12 views

Best way to handle special events at the end of each month in a time series

I am currently working with grouped time series where I have data disaggregated on a daily basis. I use Facebook Prophet and the results are in general okay with one small catch. There is one aspect ...
0
votes
1answer
15 views

Relate failure detection rate (FDR) and false alarm rate (FAR) to precision-recall

Failure detection rate (FDR) and false alarm rate (FAR) are used in anomaly detection and failure detection domains to evaluate the classification model performance. However, I don't see any clear ...
0
votes
1answer
37 views

Why does auto-encoder best suited for the job of anomaly detection?

Currently, I have been studying auto-encoders. What I have understood is that an auto-encoder is a neural network where the input layer is identical to the output layer, and it does this by minimizing ...
3
votes
1answer
51 views

Theoretical reasoning for PCA effectiveness

The effectiveness of Principal component analysis in detecting process abnormalities requires that the process data to follow a Gaussian distribution This is what I read over a thread. Can anyone ...
0
votes
0answers
23 views

Anomaly Detection in Highly Variable Time-Series Data

I am trying to detect anomalies through a column called count. The data is a time-series data and it is present for every 5 minutes for each day. The dataframe looks like this: ...
2
votes
2answers
146 views

Optimal window size for contextual outlier detection

I am looking for methods to detect univariate contextual outliers in time series data. One example application is data from industrial plants in different (unknown) operation modes or slow trends or ...
0
votes
0answers
11 views

Selecting correlated variables for time-series analysis

I am using time-series analysis to observe the spending patterns of individuals and to detect any possible anomalies, which could be fraudulent transactions. For the time-series model, I am ...
0
votes
0answers
13 views

How to make a Bayesian adaptation of a null hypothesis test?

I am trying to make software to detect anomalies from our instruments. We have a pair of instruments that each measure the same quantity but in a different way. Both instruments report a probability ...
0
votes
0answers
10 views

How can I generate a plot of the partitions in Isolation Forests

I have seen this plot is used to indicated how anomalies are isolated via partitioning in Isolation Forests. Is there a library to automatically plot this from a dataset? The plot I want to generate ...
-2
votes
1answer
40 views

How can I build AutoEncoder for a-one-class unsupervised classification model?

I am studying AutoEncoder to learn how to build a-one-class classification model which is unsupervised learning and I am wondering how to build a-one-class classification model using AutoEncoder. ...
1
vote
0answers
123 views

How can we introduce an anomaly class as a positive class to sklearn IsolationForest?

I inspired by this notebook, and I'm experimenting IsolationForest (IF) algorithm using scikit-learn==0.22.2.post1 for anomaly ...
0
votes
0answers
25 views

Autoencoder failed to find anomaaly

We're using the following architecture for anomaly-finding- ...
2
votes
1answer
49 views

Anomaly detection and explanation

I was given a dataset with 500 features which, after one-hot encoding, looks like this: Class = 1 means "anomaly", class = 0 is "normal". So basically my task is simple ML ...
1
vote
0answers
19 views

How to model a new short term beaviour in time series using ML approaches?

I had been modeling time series which contain many SKUs , due to corona there was a short-term drift in seasonality. I want to know how can i deal with this problem. Usually, sales are high in April, ...
0
votes
0answers
21 views

How to calculate f1 score for anomaly detection with one class (Anomaly or No Anomaly)? What is seen as the true positive?

How do we calculate the f1 score in anomaly detection (using a One-Class-Support Vector Machine(OC-SVM))? I am not sure what is considered as a true positive? Is it if I predict an anomaly and the ...
0
votes
1answer
110 views

incorrect results of IsolationForest

I inspired by this notebook, and I'm experimenting IsolationForest algorithm for anomaly detection context on the SF version of KDDCUP99 dataset, including 4 ...
1
vote
0answers
26 views

outlier detection for sparse data in categorical variable

I have a big dataset with a column "clientid" and a categorical column "choice". I want to find out what are the clients that have strange combinations of choices (less frequent ...
0
votes
1answer
26 views

Detecting anomalies question

This would be a data cleaning question, but I guess there are many related phrases and for sure one of them may be anomaly detection. If I have a single feature say height of humans. Question: If I ...
0
votes
0answers
28 views

'False Positive' detection in corona testing using ML techniques

I am working on the detection of false positive test result in COVID-19 testing. The false positive sample are mainly due to the contamination during the routine testing. Samples are handled in batch ...
2
votes
0answers
40 views

Step Detection (Changepoint / Anomaly)

The problem I am trying to solve is detecting step changes in a multivariate dataset, where each feature is the market share for a certain category of companies. The data is not very noisy so I would ...
0
votes
0answers
13 views

Suppressing false alarms with capturing information from unstructured corpus

In our team, we had previously deployed a machine learning anomaly detection tool at a chemical plant. It has been observed in certain cases that ongoing manual operations/interventions at plant can ...
0
votes
0answers
35 views

Anomaly detection methods assessed with AUC (poor performance unbalanced data)

I have a data with 10 000 rows and 20 columns. I also have a variable indicating if the row is an anomaly (1) or if it is a "normal data" (0). In my data there are 5% anomalies. The purpose ...
0
votes
0answers
24 views

Bayesian model for multivariate categorical observations

Context I have $n$ observations $\mathbf{X} = [\mathrm{x}_1, \dots \mathrm{x}_n]^\mathrm{T}$ from a random variable $X=(X_1, \dots X_d)$. Each variable $X_i$ is a categorical random variable with $k_i$...
0
votes
0answers
9 views

ML algorithms for classifying sequences

The problem I'm trying to solve is that of analyzing a user's behavior (for identification) from data collected through their phones. For example, accelerator, gyroscope, etc. And at any point, I have ...
0
votes
0answers
18 views

Interquartile range to detect anomaly for quarterly data

To detect anomaly for monthly data, I calculate interquartile range using the previous 12 months of data and check the current month data against that range - if the data is presented quarterly, can I ...
0
votes
0answers
54 views

How to visualise Anomaly Features

I am using an Isolation Forest Model to detect tabular data anomalies, as such, I have used shap values to identify the particular features that contribute to each anomaly. Now I would like to ...
0
votes
1answer
13 views

Using an Autoencoder for Unsupervised Outlier Detection - Train and Predict on Same Data?

I have a high-dimensional dataset of medical utilization for a public health plan's membership in which I would like to identify outliers. i.e., which individuals are potentially over- and/or under-...
0
votes
0answers
61 views

Unsupervised anomaly detection on multivariate time series

Me and my team face the next use case: Our data consists of 3 numerical signals, Which are collected every 10 minutes. Example: Our main goal is to build an anomaly detection algorithm. Our work till ...
0
votes
0answers
22 views

Testing for differences in change between multiple time distributions after an intervention

I have a dataset which contains the distribution of session lengths. For each time value it has the number of sessions that lasted for that many minutes. The data follows a logarithmic distribution. E....
2
votes
1answer
40 views

Example of a k-dimensional random vector X where each component of the vector is normally distributed but X is not [duplicate]

From the definition of multivariate normal distribution, we know that if a k-dimensional random vector X = (X1, X2, ..., Xk) is (multi-variate) normally distributed if every linear combination of its ...
0
votes
0answers
18 views

When using multi-variate gaussian distribution modelling for anomaly detection, what are the requirements on the data?

I have been studying Prof. Andrew Ng's ML course. He introduces univariate normal distribution and multivariate normal distribution as ways to model data for anomaly detection. There is a comparison ...
1
vote
0answers
59 views

Benford's law for categorical variables?

I have a dataset on avoided maritime accidents (near-miss) that looks like this: [ All variables are categorical (type=1-3, position=1-5, area=1-5, risk=1-7, 4 columns, 525 rows - every row is 1 near ...
0
votes
0answers
11 views

Outlier detection in sequential data (operational event logs)

I am currently working on a project where we want to detect outliers in sequential data (operational event logs). The first part of the conceptual method so far is as follows: Collect unlabelled ...
2
votes
0answers
35 views

LSTM architecture for anomaly detection

I'm testing out different implementation of LSTM autoencoder on anomaly detection on 2D input. My question is not about the code itself but about understanding the underlying behavior of each network. ...
0
votes
0answers
21 views

How to get more accurate parameter estimation with few samples?

Context: I have a time series on which I've trained an autoencoder model for anomaly detection (AD) purposes. My time-series is 6 years worth of daily data, but for AD purposes, I am looking at one ...
0
votes
0answers
20 views

Combining SARIMA and STL models

Can we integrate SARIMA and STL for anomaly detection? If so, what is the process like? Which model do we run first and what input goes into the other model? I am fairly new to time series so pardon ...
0
votes
0answers
45 views

Standardizing Multiple Multivariate Time Series

I have a set of devices that I am using to collect data, each device collects a multivariate time series at the same sampling rate. (around 10 minutes). I have done first-order differencing on the ...
0
votes
1answer
74 views

What is the-state-of-art for unsupervised Anomaly models through unlabeled data regarding evaluation/validation in 2020?

I'm researching anomaly detection, which is nothing else than outliers detection on a set of time-series web servers access log data or network traffic. Since outlier detection is commonly considered ...
0
votes
0answers
41 views

Multiple Time Series Anomaly Detection

I have several different unlabeled multivariate time series, each drawn from what I believe to be the same distribution, but virtually independent from each other. Each has a different length (in time)...
1
vote
1answer
44 views

Is it better to do per-class anomaly detection on P(x, y) or P(x | y)?

(Not an expert in anomaly detection.) I'd like to experiment with per-class anomaly detection. That is, we have a feature vector $x$, and a classifier that predicts its class $\hat{y}$. I'd like to ...
0
votes
0answers
366 views

Autoencoders For Multivariate Time-series Anomaly Detection

I have a multivariate time series of size (1e6, 15) and would like to fit a LSTM autoencoder. I prepare data with multivariate rolling windows (one step rolling) where each sample has (1, 5, 15) ...
0
votes
0answers
22 views

Anomaly detection for multiple correlated variables (sensors)

I have time series data from multiple sensors detecting vehicles driving through the sensor zone. I'm aggregating the total actuations for each sensor in 15-minute bins. The sensors are correlated, ...
1
vote
2answers
78 views

Decision trees for anomaly detection

Problem From what I understand, a common method in anomaly detection consists in building a predictive model trained on non-anomalous training data, and perform anomaly detection using the error of ...
0
votes
0answers
9 views

On the “1 v 49” experiment of masquerading detection data by Maxion and Townsend (2002)

I am trying to understand the definition of 1 vs 49 experiment introduced by Maxion and Townsend (2002) (Link to the paper). In short, this paper is about detecting masquerading users from the time ...
0
votes
0answers
63 views

95 % Quantile for anomaly detection

I'm building a model to detect anomalies in for multivariate data. At first stage I run VAR model every 10 min that predicts the next values, and when I get the values I calculate the Euclidean ...

1
2 3 4 5
8