Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

Anomaly detection through distribution-based hypothesis testing

I have a table of event logs that contains several categorical variables (gender, age bucket, city of residence, and education level), and I'd like to retroactively identify if a given hour of logs ...
0
votes
0answers
10 views

Setting Choosing alpha for generalized extreme Studentized deviate ESD

I'm working on a S-H-ESD implementation and I'm struggling to set the alpha for the ESD. The suggested alpha everywhere is 0.05. Is there a way to calculate an alpha based on the expected percent of ...
0
votes
0answers
14 views

How to detect the anomaly in a time series data with trend and seasonality present? [duplicate]

I want to detect the outliers in a time series data which contains the trend and seasonality components. I want to leave out the peaks which are seasonal and only consider only the other peaks. As I ...
1
vote
1answer
24 views

What outlier score is used here?

I have come across a score function in a program, but I don't exactly understand what it does. This score is a measure of how probable a sampled/created value is. I will describe the procedure for ...
-1
votes
2answers
15 views

Anomaly Detection Without a Baseline

I am attempting to find anomalies in accounting data (similar to this study: https://arxiv.org/pdf/1709.05254.pdf). I don't have any labeled data, so this attempt needs to be unsupervised. I am having ...
1
vote
0answers
26 views

Does the isolation forest care about integer-encoded categorical variables?

The isolation forest (initial paper, follow-up paper) as well as the proposed extended isolation forest (paper) seem like very appealing unsupervised anomaly detection techniques. However, the ...
1
vote
1answer
31 views

Is it OK to have only a single class labels in test data for prediction with one-class-svm?

I have a data which has only a single class, namely, '0'. There is no 'not 0' class. The one-class SVM model was trained on a <...
0
votes
0answers
59 views

Unsupervised anomaly and outlier detection of database queries

I'm monitoring database queries coming from multiple different applications spread across numerous systems and I'd like to find both anomalous queries as well as outliers in a completely unsupervised ...
2
votes
1answer
58 views

Detecting outliers in time-series if I don't have a “normal” dataset [duplicate]

I have been trying to detect anomalies in my time-series dataset. What I am trying to accomplish is the following: I have a multivariate dataset, where two columns are of special interest. One tells ...
0
votes
0answers
13 views

Usage of VIF in unsupervised model

I'm working on building an unsupervised model for real time anomaly detection based on the concept of Randomized Matrix Sketching (http://www.vldb.org/pvldb/vol9/p192-huang.pdf) which involves ...
0
votes
0answers
13 views

Can I use Free Energy as reconstruction error when I use RBM to anomaly detection?

Can I use Free Energy as reconstruction error when I use RBM to anomaly detection? If Free Energy of a sample more a threshold, can I regard it an outlier? How to explain Free Energy of RBM?
1
vote
1answer
156 views

Encoding of categorical variables with high cardinality

For unsupervised anomaly detection / fraud analytics on credit card data (where I don't have labeled fraudulent cases), there are a lot of variables to consider. The data is of mixed type with ...
0
votes
0answers
18 views

Using marginal likelihood for weighting in bayesian hierarchical model?

I have data from a series of experiments. I have a simple model for generating the data these experiments which allows me to estimate a parameter. Some experiments do not conform to my model and ...
1
vote
1answer
97 views

Estimating When A Time Series with Random Spikes Crosses a Threshold for the First Time

tl;dr Is there a way to estimate when a random spike in a time series would cross a threshold for the first time? The following is data of my performance in the game Super Hexagon, whose goal is to ...
0
votes
0answers
29 views

Unsupervised classification of images

Assuming I have a dataset of images from two similar classes, for example let's say 95% Labradors and 5% of Chihuahuas and I want to make a classifier. The point is that I need to find the anomalies (...
0
votes
0answers
16 views

Scoring the difference between a family of distributions and a test distribution

Let's suppose we have a random model that I can sample to generate distributions of a certain 1D variable. I want to score the distance of a test distribution to the model in question. The distance ...
0
votes
0answers
19 views

How do I tell if the sensors that feed a Kalman filter has diverged?

I have a time varying variable $x$ that I want to estimate. I have two sensors A and B that measure $x$. I feed their measurements to a Kalman filter. Sometimes, one of the sensors degrades for a ...
1
vote
1answer
36 views

What are the anomalies/fault/outliers detection algorithms

I'm working on a weather application that uses data coming from multiple sensors in real time (the data is time series), i've made an anomalies detection model using One Class Support Vector Machines, ...
0
votes
0answers
39 views

Chi-Square Degeneracy for Large Sample

(Forgive my hand-waving explanation) When discussing anomaly detection methods (for example), one possibility is comparing the distance of a point from a centroid: Given 100 samples $X_1,...,X_{100}$ ...
0
votes
1answer
56 views

Anomaly detection in time series data from multiple sensors [closed]

I've build a classification model based on 15 features coming in real time from 15 sensors. The window time is 60 seconds, means that the classification model needs 60 records from each sensor (the ...
0
votes
0answers
18 views

Algorithm to detect outliers in network sensor messages

I have a network sensor device which generates a number of messages. The message is of format "timeofmessage messagetype messageimportance messagetext". The sensor keeps producing "sensor-ok" messages ...
3
votes
2answers
55 views

How to identify outliers in a time series with correlated variables

I am working with time series data of sensor measurements. I have nine sensors that are in the same ballpark location recording the same data every 10 minutes. The sensors are setup such that the ...
1
vote
0answers
35 views

software for one class classification with a Bayesian Network

I'm looking for a software package that would allow to do a one class classification with a Bayesian Network (anomaly detection). I was planning to use bnlearn but so far I'm unable to find out if ...
0
votes
0answers
31 views

EDA of Time Series Data (System Logs)

My project is about exploring time series data - system logs of all desktops in a large organisation. The idea is to see whether this data can be used for: (A) diagnostics (B) anomaly detection The ...
0
votes
0answers
13 views

Newbie wondering about standard deviation and how to detect outliers in new datapoints

So I have a dataset, with the count of records processed per day in each, e.g. | 27th April: 3491 | 28th April: 2058 | 29th April: 9321 | 30th April: 1021 I want to be able to take a new day's count ...
0
votes
0answers
23 views

How to find anomalies/outliers in Panel Data (Unsupervised)?

I have panel data based on 900000 different entities with 384 time steps and the data is not normally distributed. I am looking for outliers/anomalies, this is unsupervised as I have no examples of ...
0
votes
1answer
62 views

How to get top features that contribute to anomalies in Isolation forest

I am using Isolation forest for anomaly detection on multidimensional data. The algorithm is detecting anomalous records with good accuracy. Apart from detecting anomalous records I also need to find ...
0
votes
1answer
35 views

When does it make sense to detect multivariate outliers instead of univariate ones?

I do get the idea of univariate outliers and detecting them. However, I don't understand the idea of multivariate outliers. More precisely, I would like to ask if detecting multivariate outliers only ...
0
votes
1answer
24 views

Modelling small data set problem

I have a small dataset (20 instances per 13 classes). The 13 classes are human users from their behavior features, I have to classify if an unseen behavior feature is of a user or not. Data: These ...
1
vote
0answers
32 views

anomaly detection : check the separability of normal and abnormal data

I'd like to develop an anomaly detection. I have historical data from sensors in the form of time series. The time series can be divided into data of a normal state and data of an abnormal state i.e. ...
0
votes
1answer
26 views

How to find categorical contributing factors for an anomaly?

Given a house sales dataset with number of houses sold each day and their attributes (i.e., price, number of rooms, size, etc.) - if on a specific day there's a spike/drop in sales, what are some ...
3
votes
2answers
64 views

Strange encoding for categorical features

I am reading through https://arxiv.org/pdf/1609.06676.pdf which presents an extension of the isolation forest algorithm so that categorical features may be taken into account. On page 5, the authors ...
0
votes
0answers
41 views

Using exponential smoothing with modified z score

I am finding anomalies in my data. My data contains exit rate of a website for the last 3 years with a daily frequency. When using just modified Z score & finding anomalies, it does not take ...
0
votes
0answers
24 views

test of significance for anomaly detection

I am using an anomaly detection algorithm (Twitter's Anomaly Detection) that uses SH-ESD method. I 'm finding anomalies in a time-series of the daily counts. I can obtain TP, FP and FN measures. Is ...
0
votes
0answers
42 views

Scalability of Tsoutlier

I am currently using TSO() in R to find the anomalies. I am doing this for last 3 years daily data. I am getting the output as well with list of anomalies. If i need to repeat on a daily basis with ...
0
votes
0answers
7 views

How to combine KMeans+CART for network anomalies detection in KDD 99

I am looking to detect anomalies in the KDD 99 data using both the Kmeans and the CART decision tree. The objective is to show that accurence are better when using the single kmeans. Thank you in ...
1
vote
1answer
147 views

Anomaly detection with complex seasonality

I am working on a anomaly detection for a batches of daily time series (non-hierarchical) that exhibit both yearly and weekly seasonality. I tested a few algorithms and it appears that tbats() from ...
0
votes
0answers
16 views

Could a graph with a lot of noise be classified as anomalies?

I am studying the below plot and trying to identify if there are any anomalies. Anomalies, as I understand, is synonymous with outliers. The x axis represents the number of followers for some social ...
1
vote
1answer
21 views

Operationalizing a problem: Time-series Anomaly detection with multiple units and multiple sensors

I have an IoT problem I am trying to operationalize. I have multiple machines that should behave similarly over time (a good example is wind turbines nearby). They each have multiple sensors. And I ...
0
votes
1answer
18 views

How can i know that my dataset is being well distributed with K-means?

I'm trying to make an anomaly detection system using Spark Mlib an its K-means implementation but i'm struggling to decide when should i stop searching for K. I'm following Chapter 5 of the Advanced ...
1
vote
1answer
805 views

Feature Importance in Isolation Forest

In an unsupervised setting for higher-dimensional data (e.g. 10 variables (numerical and categorical), 5000 samples, ratio of anomalies likely 1% or below but unknown) I am able to fit the isolation ...
0
votes
0answers
45 views

LocalOutlierFactor scikit-learn

My goal is to use the LocalOutlierFactor class from scikit-learn to do real-time Novelty Detection. This can be achieved by setting novelty=True in the constructor, ...
1
vote
1answer
802 views

Can One-Class SVM be used for outlier detection?

According to my readings (Support Vector Method for Novelty Detection, for instance), One-Class SVM can be used for novelty detection only. The purpose of the $\nu$ parameter is to defined the maximum ...
0
votes
0answers
154 views

Adaptive threshold setting for parametric anomaly detection system applied to time series data

I just started my first project where I'm trying to find anomalies in the energy usage of a air conditioner. The only usable data I could obtain was the energy data for a few months. Since the energy ...
0
votes
0answers
27 views

Decomposition Difference Between Twitter and STL Method

I am having a lot of trouble understanding the difference between the two decomposition methods: twitter and stl. https://www.rdocumentation.org/packages/anomalize/versions/0.1.1/topics/...
0
votes
0answers
86 views

Anomalize Package time_decompose

I am trying to figure out what the meaning is behind each of the different compositions of the time_decompose function in the ...
1
vote
1answer
97 views

DeepLearning & Anomaly Detection - Understanding & How to Properly Tune

I'm looking into understanding the Deeplearning anomaly detection algorithm provided by h2o. I tried to recreate an example below. Perhaps some of these questions are basic, but I'm trying to better ...
1
vote
0answers
20 views

Use internal representation of autoencoder for anomaly detection [closed]

I've trained an autoencoder to recognize 'positive' time series (the network is a simple fully connected network, no recurrent layers). The problem is that from what my advisor says, I should try to ...
1
vote
1answer
1k views

Anomaly detection in multivariate time series data

I am trying to solve an anomaly detection problem that consists of three variables captured over a span of five years. It is an unsupervised problem, and I believe density-based clustering methods ...
2
votes
0answers
153 views

Benchmark data sets for anomaly detection algorithms in multivariate time series

as per title, which datasets are commonly used to benchmark novel methods to detect anomalies in multivariate time series? I'm particularly interested in moderately high-dimensional (10-40 components),...