Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

How to describe a period based on anomalies?

I have a raster stack with temperature anomalies. Every raster represents a year and I want to find a metric to "summarise" all the anomalies in one raster object. What metric should I use to better ...
0
votes
0answers
9 views

Identifying anomalies multiple time series using statistical methods

I am trying to detect anomalies from a Time Series sequence, that is calculated as an aggregate count of multiple unique Time Series'. For example, in the graph below: We have aggregate counts shown ...
2
votes
0answers
21 views

One-Class SVM - Decision function

The following is based on the paper: Schölkopf et.al - SVM for Novelty Detection First let us consider the (classical) Soft Margin SVM optimization problem: ${\displaystyle {\text{minimize }}{\frac {...
0
votes
1answer
33 views

Method(s) to predict binary outcome from onset of variation in time series data

Context: I have data on 100 patients showing their time of attendance at a service. They attend on a roughly daily basis between 9-5 (except weekends and occasional missed appointments). They access a ...
0
votes
2answers
58 views

Outliers Detection with unlabeled data? [closed]

I have a dataframe with numeric and categorical variables and no target variables and I need to check for multivariate outliers. Could you suggest a model (using Python) that works good for outliers ...
0
votes
0answers
20 views

Does it make sense to validate sales data using RSD?

Here's a nonsense example of data I'm working with: ...
0
votes
0answers
18 views

Explain autoencoder anomalies

I developed an autoencoder model to detect anomalies in a set of signals coming from a machine. After the scoring, the most anomalous point (i.e. the ones with highest reconstruction error) are ...
-1
votes
1answer
111 views

Is time series analysis suitable for my dataset?

I am monitoring user behavior, while the user interacts with a form on a website. That form has multiple textfields from top to bottom and at the bottom it has two buttons: "cancel" and "save". My ...
1
vote
1answer
55 views

How to use PCA to detect outliers?

A PCA will reduce the dimensionality of the original data and construct a subspace generated by eigenvectors of which each represents the (next) highest variance to explain the data. Let's start at ...
0
votes
1answer
85 views

Autoencoder reconstruction error threshold

I have a set of signals on which I have to implement an anomaly detection algorithm. The data is split among a reference period (i.e. last 3 months) and a test period (i.e. last week). I've already ...
1
vote
2answers
89 views

Simple outlier detection for time series

I wanted to generate a very simple example of anomaly detection for time series. So I created sample data with one very obvious outlier. Here's a picture of the data: The problem is, I didn't get any ...
0
votes
1answer
55 views

Isolation forest with categorical data?

I understand how isolation forests can work with numeric data, but I wonder how it can work with categorical data? Also, at least when working with Sci-kit-Learn, the recommendation I saw was to ...
0
votes
1answer
29 views

What is the difference between sklearn IsolationForests score_samples and decision_function?

The predict method will output -1 (anomaly) where forest.decision_function(X) < forest.threshold_ and 1 otherwise. But what does ...
0
votes
1answer
47 views

Machine Learning alternative for hashing [closed]

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
0
votes
0answers
25 views

Detect anomaly users who try to access too often based on the access log

Does anyone give me advice for statistically detecting anomaly users who try to log in our website too often? At first, the idea that came to my mind is to use Spike detection approaches or IQR ...
2
votes
1answer
56 views

Isolation Forest and average/expected depth formula

The Isolation Forest algorithm (Liu, Fei Tony, Kai Ming Ting, and Zhi-Hua Zhou. "Isolation forest." 2008 Eighth IEEE International Conference on Data Mining. IEEE, 2008. - link: https://cs.nju.edu.cn/...
0
votes
0answers
16 views

how to detect ouliers in skewed data(left skewed distribution) [duplicate]

I have studied a lot of ways of dealing with outliers of normal or multidimensional data. But my problem is about skewed data. How can I find outliers for data with a skewed distribution to the left?
0
votes
0answers
41 views

Interpreting Local Outlier Factor (LOF) results

Using this example on the scikit-learn site, I am attempting to do some anomaly detection using LOF. What I end up with is this: ...
3
votes
2answers
71 views

anomaly detection in time series training data

I have a dataset which basically is measuring the number of people passing a certain region which is monitored and I basically have these raw counts of people over the last two months at 5 minutes ...
0
votes
0answers
22 views

Find plausible peaks in streamed data

i have got a signal of a streamed source which produces values like in the picture. I want to get the "real" peaks (blue circles). But the noisy peaks (red circles) mess up the peak search. The ...
0
votes
0answers
24 views

Unsupervised binary classification with mixed data: Anomaly detection?

I apologize for the lengthy question, but this problem has been troubling me for quite some time now and I can't seem to find an answer to another question which directly applies to my situation. I ...
0
votes
2answers
46 views

Supervised anomaly detection of multiple time series

I'd like to develop a set of models for anomaly detection of multiple time series. After some reading, I have found a few promising approaches, such as Segmentation-based approaches (SECODA); ...
1
vote
1answer
32 views

How detrend a line if i know the slope? [closed]

I am using linear regression to get the slope of some data. If i know the slope how can i flatten the line so that it has no slope?
0
votes
0answers
14 views

Robust way to calculate anomalies of short time series

My question is not related to any actual piece of code, but rather with the thought behind it. I have a short time series of about 20 years of climate data and I want to calculate anomalies. Usually,...
1
vote
2answers
31 views

scikit-learn IsolationForest no variance feature

I'm using IsolationForest algorithm in order to detect anomalies in my data and to use this model to detect future anomalies in new rows and came across a few questions: Is the model good for ...
0
votes
1answer
32 views

Clarification in Anomaly Detection Algorithm

I am referring to Prof Andrew Ng Coursera ML notes (Week 9). He says that to identify outliers we first model the training data and then fit a Gaussian distribution with probability density $ p(X; \mu ...
0
votes
0answers
17 views

Help in Handling multiple classes in independent categorical variables and improve performance

The dataset has 4 categorical and 1 numerical variable and a timestamp variable. Out of 4, three categorical variables are having more than 100 categories. I tried doing one-hot encoding on the whole ...
0
votes
0answers
17 views

Anomaly detection through distribution-based hypothesis testing

I have a table of event logs that contains several categorical variables (gender, age bucket, city of residence, and education level), and I'd like to retroactively identify if a given hour of logs ...
0
votes
0answers
18 views

Setting Choosing alpha for generalized extreme Studentized deviate ESD

I'm working on a S-H-ESD implementation and I'm struggling to set the alpha for the ESD. The suggested alpha everywhere is 0.05. Is there a way to calculate an alpha based on the expected percent of ...
0
votes
0answers
15 views

How to detect the anomaly in a time series data with trend and seasonality present? [duplicate]

I want to detect the outliers in a time series data which contains the trend and seasonality components. I want to leave out the peaks which are seasonal and only consider only the other peaks. As I ...
1
vote
1answer
30 views

What outlier score is used here?

I have come across a score function in a program, but I don't exactly understand what it does. This score is a measure of how probable a sampled/created value is. I will describe the procedure for ...
0
votes
2answers
25 views

Anomaly Detection Without a Baseline

I am attempting to find anomalies in accounting data (similar to this study: https://arxiv.org/pdf/1709.05254.pdf). I don't have any labeled data, so this attempt needs to be unsupervised. I am having ...
1
vote
0answers
108 views

Does the isolation forest care about integer-encoded categorical variables?

The isolation forest (initial paper, follow-up paper) as well as the proposed extended isolation forest (paper) seem like very appealing unsupervised anomaly detection techniques. However, the ...
1
vote
1answer
35 views

Is it OK to have only a single class labels in test data for prediction with one-class-svm?

I have a data which has only a single class, namely, '0'. There is no 'not 0' class. The one-class SVM model was trained on a <...
0
votes
0answers
63 views

Unsupervised anomaly and outlier detection of database queries

I'm monitoring database queries coming from multiple different applications spread across numerous systems and I'd like to find both anomalous queries as well as outliers in a completely unsupervised ...
2
votes
1answer
65 views

Detecting outliers in time-series if I don't have a “normal” dataset [duplicate]

I have been trying to detect anomalies in my time-series dataset. What I am trying to accomplish is the following: I have a multivariate dataset, where two columns are of special interest. One tells ...
0
votes
0answers
14 views

Usage of VIF in unsupervised model

I'm working on building an unsupervised model for real time anomaly detection based on the concept of Randomized Matrix Sketching (http://www.vldb.org/pvldb/vol9/p192-huang.pdf) which involves ...
0
votes
1answer
26 views

Can I use Free Energy as reconstruction error when I use RBM to anomaly detection?

Can I use Free Energy as reconstruction error when I use RBM to anomaly detection? If Free Energy of a sample more a threshold, can I regard it an outlier? How to explain Free Energy of RBM?
1
vote
1answer
1k views

Encoding of categorical variables with high cardinality

For unsupervised anomaly detection / fraud analytics on credit card data (where I don't have labeled fraudulent cases), there are a lot of variables to consider. The data is of mixed type with ...
0
votes
0answers
19 views

Using marginal likelihood for weighting in bayesian hierarchical model?

I have data from a series of experiments. I have a simple model for generating the data these experiments which allows me to estimate a parameter. Some experiments do not conform to my model and ...
1
vote
1answer
126 views

Estimating When A Time Series with Random Spikes Crosses a Threshold for the First Time

tl;dr Is there a way to estimate when a random spike in a time series would cross a threshold for the first time? The following is data of my performance in the game Super Hexagon, whose goal is to ...
0
votes
0answers
36 views

Unsupervised classification of images

Assuming I have a dataset of images from two similar classes, for example let's say 95% Labradors and 5% of Chihuahuas and I want to make a classifier. The point is that I need to find the anomalies (...
0
votes
0answers
22 views

Scoring the difference between a family of distributions and a test distribution

Let's suppose we have a random model that I can sample to generate distributions of a certain 1D variable. I want to score the distance of a test distribution to the model in question. The distance ...
0
votes
0answers
25 views

How do I tell if the sensors that feed a Kalman filter has diverged?

I have a time varying variable $x$ that I want to estimate. I have two sensors A and B that measure $x$. I feed their measurements to a Kalman filter. Sometimes, one of the sensors degrades for a ...
1
vote
1answer
40 views

What are the anomalies/fault/outliers detection algorithms

I'm working on a weather application that uses data coming from multiple sensors in real time (the data is time series), i've made an anomalies detection model using One Class Support Vector Machines, ...
0
votes
0answers
40 views

Chi-Square Degeneracy for Large Sample

(Forgive my hand-waving explanation) When discussing anomaly detection methods (for example), one possibility is comparing the distance of a point from a centroid: Given 100 samples $X_1,...,X_{100}$ ...
0
votes
1answer
97 views

Anomaly detection in time series data from multiple sensors [closed]

I've build a classification model based on 15 features coming in real time from 15 sensors. The window time is 60 seconds, means that the classification model needs 60 records from each sensor (the ...
0
votes
0answers
18 views

Algorithm to detect outliers in network sensor messages

I have a network sensor device which generates a number of messages. The message is of format "timeofmessage messagetype messageimportance messagetext". The sensor keeps producing "sensor-ok" messages ...
3
votes
2answers
88 views

How to identify outliers in a time series with correlated variables

I am working with time series data of sensor measurements. I have nine sensors that are in the same ballpark location recording the same data every 10 minutes. The sensors are setup such that the ...
1
vote
0answers
39 views

software for one class classification with a Bayesian Network

I'm looking for a software package that would allow to do a one class classification with a Bayesian Network (anomaly detection). I was planning to use bnlearn but so far I'm unable to find out if ...