Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. This is also known as outlier detection.

0
votes
0answers
13 views

How do I tell if the sensors that feed a Kalman filter has diverged?

I have a time varying variable $x$ that I want to estimate. I have two sensors A and B that measure $x$. I feed their measurements to a Kalman filter. Sometimes, one of the sensors degrades for a ...
1
vote
1answer
32 views

What are the anomalies/fault/outliers detection algorithms

I'm working on a weather application that uses data coming from multiple sensors in real time (the data is time series), i've made an anomalies detection model using One Class Support Vector Machines, ...
0
votes
0answers
19 views

Chi-Square Degeneracy for Large Sample

(Forgive my hand-waving explanation) When discussing anomaly detection methods (for example), one possibility is comparing the distance of a point from a centroid: Given 100 samples $X_1,...,X_{100}$ ...
0
votes
1answer
31 views

Anomaly detection in time series data from multiple sensors [closed]

I've build a classification model based on 15 features coming in real time from 15 sensors. The window time is 60 seconds, means that the classification model needs 60 records from each sensor (the ...
0
votes
0answers
29 views

Detecting seasonal anomalies using k-means [closed]

I have a huge network log file which contains messages from all different devices in the LAN network. A number of devices send periodic/cyclic messages - some messages are sent every hour, some are ...
0
votes
0answers
17 views

Algorithm to detect outliers in network sensor messages

I have a network sensor device which generates a number of messages. The message is of format "timeofmessage messagetype messageimportance messagetext". The sensor keeps producing "sensor-ok" messages ...
3
votes
2answers
37 views

How to identify outliers in a time series with correlated variables

I am working with time series data of sensor measurements. I have nine sensors that are in the same ballpark location recording the same data every 10 minutes. The sensors are setup such that the ...
1
vote
0answers
24 views

software for one class classification with a Bayesian Network

I'm looking for a software package that would allow to do a one class classification with a Bayesian Network (anomaly detection). I was planning to use bnlearn but so far I'm unable to find out if ...
0
votes
0answers
20 views

EDA of Time Series Data (System Logs)

My project is about exploring time series data - system logs of all desktops in a large organisation. The idea is to see whether this data can be used for: (A) diagnostics (B) anomaly detection The ...
0
votes
0answers
13 views

Newbie wondering about standard deviation and how to detect outliers in new datapoints

So I have a dataset, with the count of records processed per day in each, e.g. | 27th April: 3491 | 28th April: 2058 | 29th April: 9321 | 30th April: 1021 I want to be able to take a new day's count ...
0
votes
0answers
8 views

How to find anomalies/outliers in Panel Data (Unsupervised)?

I have panel data based on 900000 different entities with 384 time steps and the data is not normally distributed. I am looking for outliers/anomalies, this is unsupervised as I have no examples of ...
0
votes
0answers
16 views

How to get top features that contribute to anomalies in Isolation forest

I am using Isolation forest for anomaly detection on multidimensional data. The algorithm is detecting anomalous records with good accuracy. Apart from detecting anomalous records I also need to find ...
0
votes
1answer
26 views

When does it make sense to detect multivariate outliers instead of univariate ones?

I do get the idea of univariate outliers and detecting them. However, I don't understand the idea of multivariate outliers. More precisely, I would like to ask if detecting multivariate outliers only ...
0
votes
1answer
21 views

Modelling small data set problem

I have a small dataset (20 instances per 13 classes). The 13 classes are human users from their behavior features, I have to classify if an unseen behavior feature is of a user or not. Data: These ...
1
vote
0answers
28 views

anomaly detection : check the separability of normal and abnormal data

I'd like to develop an anomaly detection. I have historical data from sensors in the form of time series. The time series can be divided into data of a normal state and data of an abnormal state i.e. ...
0
votes
1answer
20 views

How to find categorical contributing factors for an anomaly?

Given a house sales dataset with number of houses sold each day and their attributes (i.e., price, number of rooms, size, etc.) - if on a specific day there's a spike/drop in sales, what are some ...
1
vote
0answers
26 views

Strange encoding for categorical features

I am reading through https://arxiv.org/pdf/1609.06676.pdf which presents an extension of the isolation forest algorithm so that categorical features may be taken into account. On page 5, the authors ...
0
votes
0answers
30 views

Using exponential smoothing with modified z score

I am finding anomalies in my data. My data contains exit rate of a website for the last 3 years with a daily frequency. When using just modified Z score & finding anomalies, it does not take ...
0
votes
0answers
16 views

test of significance for anomaly detection

I am using an anomaly detection algorithm (Twitter's Anomaly Detection) that uses SH-ESD method. I 'm finding anomalies in a time-series of the daily counts. I can obtain TP, FP and FN measures. Is ...
0
votes
0answers
42 views

Scalability of Tsoutlier

I am currently using TSO() in R to find the anomalies. I am doing this for last 3 years daily data. I am getting the output as well with list of anomalies. If i need to repeat on a daily basis with ...
0
votes
0answers
6 views

How to combine KMeans+CART for network anomalies detection in KDD 99

I am looking to detect anomalies in the KDD 99 data using both the Kmeans and the CART decision tree. The objective is to show that accurence are better when using the single kmeans. Thank you in ...
1
vote
1answer
70 views

Anomaly detection with complex seasonality

I am working on a anomaly detection for a batches of daily time series (non-hierarchical) that exhibit both yearly and weekly seasonality. I tested a few algorithms and it appears that tbats() from ...
0
votes
0answers
16 views

Could a graph with a lot of noise be classified as anomalies?

I am studying the below plot and trying to identify if there are any anomalies. Anomalies, as I understand, is synonymous with outliers. The x axis represents the number of followers for some social ...
1
vote
1answer
17 views

Operationalizing a problem: Time-series Anomaly detection with multiple units and multiple sensors

I have an IoT problem I am trying to operationalize. I have multiple machines that should behave similarly over time (a good example is wind turbines nearby). They each have multiple sensors. And I ...
0
votes
1answer
18 views

How can i know that my dataset is being well distributed with K-means?

I'm trying to make an anomaly detection system using Spark Mlib an its K-means implementation but i'm struggling to decide when should i stop searching for K. I'm following Chapter 5 of the Advanced ...
0
votes
1answer
476 views

Feature Importance in Isolation Forest

In an unsupervised setting for higher-dimensional data (e.g. 10 variables (numerical and categorical), 5000 samples, ratio of anomalies likely 1% or below but unknown) I am able to fit the isolation ...
0
votes
0answers
36 views

LocalOutlierFactor scikit-learn

My goal is to use the LocalOutlierFactor class from scikit-learn to do real-time Novelty Detection. This can be achieved by setting novelty=True in the constructor, ...
1
vote
1answer
512 views

Can One-Class SVM be used for outlier detection?

According to my readings (Support Vector Method for Novelty Detection, for instance), One-Class SVM can be used for novelty detection only. The purpose of the $\nu$ parameter is to defined the maximum ...
0
votes
0answers
121 views

Adaptive threshold setting for parametric anomaly detection system applied to time series data

I just started my first project where I'm trying to find anomalies in the energy usage of a air conditioner. The only usable data I could obtain was the energy data for a few months. Since the energy ...
0
votes
0answers
26 views

Decomposition Difference Between Twitter and STL Method

I am having a lot of trouble understanding the difference between the two decomposition methods: twitter and stl. https://www.rdocumentation.org/packages/anomalize/versions/0.1.1/topics/...
0
votes
0answers
63 views

Anomalize Package time_decompose

I am trying to figure out what the meaning is behind each of the different compositions of the time_decompose function in the ...
1
vote
1answer
86 views

DeepLearning & Anomaly Detection - Understanding & How to Properly Tune

I'm looking into understanding the Deeplearning anomaly detection algorithm provided by h2o. I tried to recreate an example below. Perhaps some of these questions are basic, but I'm trying to better ...
1
vote
0answers
18 views

Use internal representation of autoencoder for anomaly detection [closed]

I've trained an autoencoder to recognize 'positive' time series (the network is a simple fully connected network, no recurrent layers). The problem is that from what my advisor says, I should try to ...
1
vote
1answer
912 views

Anomaly detection in multivariate time series data

I am trying to solve an anomaly detection problem that consists of three variables captured over a span of five years. It is an unsupervised problem, and I believe density-based clustering methods ...
1
vote
0answers
112 views

Benchmark data sets for anomaly detection algorithms in multivariate time series

as per title, which datasets are commonly used to benchmark novel methods to detect anomalies in multivariate time series? I'm particularly interested in moderately high-dimensional (10-40 components),...
0
votes
0answers
86 views

Which method for anomaly detection on curves of sensor data

I have data of measurements of a sensor. The sensor measures a numeric value at 20 fixed positions every few seconds. You can think of a camera traversing over a plate and measuring the amount of ...
1
vote
1answer
225 views

Matrix Profile vs. Deep Learning

I've been reading about Matrix Profiles and how they can be used for anomaly detection in time-series. However, I'm a bit confused how they compare/relate to typical Deep Learning approaches. I would ...
0
votes
0answers
177 views

Using ARIMA with exogenous regressors for outlier detection in R

I would like to detect outliers in real-time data that is aggregated per hour. For this example, I've selected the hourly pedestrian data from Melbourne, Australia (Pedestrian volume (updated monthly),...
0
votes
1answer
216 views

ELBO interpretation in Variational Autoencoder (VAE) for anomaly detection

How to interpret different ELBO values when checking anomaly detection possibilities of VAE model on different "testing" datasets? The higher the ELBO value of the model when testing it on different ...
2
votes
0answers
24 views

Ruling out collusion in peer evaluation

In my class I have groups of students making presentations and the rest of the class evaluates them. Since their scores are curved the students do have an incentive to collude and mark other teams ...
3
votes
0answers
33 views

Sequential Prediction: Data Modeling and Classical Algorithms

I have data that can be called demographic data. Raw data Person 0001 \begin{array}{|c|c|} \hline Feb\,1981- Apr\,85 & engaged\,\,in\,\,\underline{activity}\,\,\textit{A}\,\,of \,\,\underline{...
1
vote
0answers
164 views

Variable importance (?) for multivariate time series anomaly detection methods

I'm working on anomaly detection methods for multivariate time series $[\mathbf{x}^{new}_1,\dots,\mathbf{x}^{new}_T]$ where $\mathbf{x}^{new}_{i}$ is $p-$dimensional. I won't go into the details of ...
0
votes
1answer
98 views

Bounded Anomaly Score between 0 and 1

I am using a KNN anomaly detection approach, where the distance to my nearest neighbor is an indication for an anomaly. I am wondering how I can normalize the score between 0 and 1. I can use a test ...
1
vote
0answers
395 views

Feature Importances for Isolation Forest?

Is there a way to dig in to sklearn's Isolation Forest algorithm to understand why data is scored as an inlier or outlier? I see that you can produce a score with decision_function (for data used to ...
0
votes
0answers
30 views

How to classify similar looking dataset but belonging to different class

I've got a user dataset in which there are two classes. The size of dataset if 50,000. Class_A=5000 , class_B= 45000. Now the problem is that there are some instances(500) which though belong to ...
1
vote
0answers
20 views

Performance of Hierarchical Temporal Memory on unsupervised online anomaly detection problems

I'm looking for an algorithm to detect anomalies in streaming data (server metrics). The detection needs to be near-real time and unsupervised (labeled data will never be available, unfortunately, and ...
0
votes
0answers
16 views

What test should I use to compare 2 group with very similar characteristic?

I am currenty doing a fraud detection analysis. I have 2 groups of data points with several features that I create myself. 1 group is fraud people and another group is non fraud. I found out that ...
0
votes
2answers
239 views

Anomaly detection in Text Classification

I have built a text classifier using OneClassSVM. I have the training set which corresponds to only one label i.e("Yes") and I don't have the other("NO") label data. My task is to build a classifier ...
1
vote
2answers
96 views

How to choose a method for binary classifier based on only positive and unlabelled examples?

I need to build a binary classifier with machine learning, as I fail to manually choose a combination of features to achieve minimal fraction of false positives. What is best practice for choosing a ...
2
votes
0answers
382 views

Robust Anomaly Detection Algorithm from Netflix?

I have read a lot about the robust anomaly detection of Netflix which they open sourced as part of their Surus Project (https://github.com/Netflix/Surus). The project anomaly detector is based on the ...