Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0
votes
0answers
16 views

Subtracting PCA projection from original data

I'm trying to implement Anomaly Detection algorythm from this article https://iopscience.iop.org/article/10.1088/1742-6596/1069/1/012072/pdf at my work. So I have big questions for me: Did I ...
4
votes
2answers
77 views

How to find anomalies in data by using ARIMA

I want to detect anomalies in my data. It is in the red square shown in the figure. This is a person's chest movement data in 60 seconds. The red square is called "apnea", which a person stops ...
0
votes
0answers
10 views

Are there any methods to detect whole multivariate time-series as anomalous from a set of multivariate time-series?

Consider a scenario with Dataset D as {T1, T2, ..., Tn} and Ti is a multivariate time-series of length mi as {X1, X2, ..., Xmi}. Here each record of the time-series Xi is a vector of attribute values {...
2
votes
1answer
43 views

Recommendations for anomaly detection

I have a binary classifier. The classifier is trained on both numeric and categorical variables. In a given month, I will have new data coming in, comparable to 5% of the observation count of the ...
0
votes
1answer
22 views

Point anomalies vs collective anomalies in time series

The Anomaly Detection survey by Chandola et al categorizes anomalies into point anomalies and collective anomalies. Do we need this type of categorization in anomaly detection in time series, though? ...
0
votes
1answer
23 views

Detecting anomalies in natural vibration data

I'm looking into using clustering-based method to detect anomalies in vibration signals. The idea is to extract features for every single sliding time-window of a time series of normal vibration data, ...
1
vote
0answers
18 views

From unsupervised to supervised in fraud detection

I have a question. I am working on the fraud detection domain. And I have data from imports to the country. As you can get from the title, I have unsupervised data. I do not know that the record is ...
0
votes
1answer
32 views

Is autoencoder for anomaly detection a transfer learning?

I am doing a binary classification with unsupervised learning. I learn an autoencoder on samples from class0 and then predict samples from both class0 and class1. Then I classify sample according to ...
0
votes
1answer
34 views

Outlier detection using the difference between two z-scores

Long story short: Can you use the difference in z-score of two variables as an outlier detector. I have this data set which had poor quality data. Lots of measurement/human error and probably also ...
0
votes
1answer
26 views

How to deal with over-represented features in anomaly detection

Let me illustrate the problem with an example: Our dataset consists of a collection of letters written to Santa Claus mostly by kids together with the age of the person who wrote them. We want to run ...
1
vote
0answers
14 views

Detecting anomalies in system logs

Chances are this may be closed off as too broad, but I'll try to be as specific as I can. I am currently working with API logs with categorical features separated by 1ms intervals, an example: ...
0
votes
0answers
14 views

Detecting the presence outliers, while ignoring pure noise

I would like to detect peaks in a time series, but all too often a bunch of noise gets picked up as well, and fools most algorithm I throw at it. Often, to get it to work, I need to tweak the ...
0
votes
0answers
30 views

Oversampling/Undersampling in respect to Train and Test - Isolation Forest

I've got a quite imbalanced data set. 144.496 : 162 -> ratio of 1000:1 I would like to use IsolationForest to detect the 162 anomalys. I've already split the data. However, the iForest doesn't ...
0
votes
0answers
38 views

Anomaly detection using reconstruction error (compositional data)

I have compositional data (nonnegative and sum to 1) in a data matrix with 150 samples (rows) and 4096 features (columns). The data is made up of 144 normal samples and 6 known anomalies, which I'm ...
0
votes
1answer
20 views

Is there a tool that can tell me how a subset is different from a whole (in terms of features)

Is there a tool/technique where we can specify a subset of tuples (of feature values that are real numbers) from an overall set and ask how this subset could be anomalous? (i.e., what are the key ...
1
vote
2answers
15 views

How to interpret the precision score of a model when the problem is an anomaly detection?

In a binary classification with a balanced label, the precision score is usually considered “positive” if it performs better than a random guess, i.e a precision above 0.5. When the event (label = 1) ...
1
vote
1answer
46 views

Uni-variate hourly time series anomaly detection by TBATS

I have hourly and +4 years length of air pollution data (PM10). It has 24 (daily), 168 (weekly) and 8766 (yearly) seasonality. Also distribution is right skewed and has very long tail. I want to make ...
2
votes
1answer
42 views

Anomaly detection using vector autoregression

I want to detect anomalies in multivariate time series using statistical approaches. In particular. I want to use a vector autoregression model like VAR, VARMA or VARIMA, to predict a time stamp $x_t$ ...
1
vote
1answer
50 views

Trying to smooth small 'bumps' in graph data using spline interpolation for changepoint detection

I'm trying to detect changes in my data, I want to identify points that are like local minima and shoot upwards. I have used the changepoint package to do so, and upon running it and selecting my ...
0
votes
0answers
71 views

One-Class SVM for anomaly detection on time series

I want to use One-Class SVM to detect anomalies on univariate or multivariate time series. The problem is that SVM can be only applied to a set of vectors and are not aimed for time series. The paper ...
0
votes
1answer
23 views

How to use an Autoencoder for anomaly detection?

I would like to use an Autoencoder for anomaly detection and I wonder how to detect them basically. This is a general question but also related to an implementation as I use time series data. That's ...
0
votes
0answers
20 views

How to describe a period based on anomalies?

I have a raster stack with temperature anomalies. Every raster represents a year and I want to find a metric to "summarise" all the anomalies in one raster object. What metric should I use to better ...
2
votes
0answers
41 views

One-Class SVM - Decision function

The following is based on the paper: Schölkopf et.al - SVM for Novelty Detection First let us consider the (classical) Soft Margin SVM optimization problem: ${\displaystyle {\text{minimize }}{\frac {...
0
votes
1answer
38 views

Method(s) to predict binary outcome from onset of variation in time series data

Context: I have data on 100 patients showing their time of attendance at a service. They attend on a roughly daily basis between 9-5 (except weekends and occasional missed appointments). They access a ...
0
votes
2answers
111 views

Outliers Detection with unlabeled data? [closed]

I have a dataframe with numeric and categorical variables and no target variables and I need to check for multivariate outliers. Could you suggest a model (using Python) that works good for outliers ...
0
votes
0answers
21 views

Does it make sense to validate sales data using RSD?

Here's a nonsense example of data I'm working with: ...
0
votes
0answers
25 views

Explain autoencoder anomalies

I developed an autoencoder model to detect anomalies in a set of signals coming from a machine. After the scoring, the most anomalous point (i.e. the ones with highest reconstruction error) are ...
-1
votes
1answer
114 views

Is time series analysis suitable for my dataset?

I am monitoring user behavior, while the user interacts with a form on a website. That form has multiple textfields from top to bottom and at the bottom it has two buttons: "cancel" and "save". My ...
3
votes
1answer
530 views

How to use PCA to detect outliers?

A PCA will reduce the dimensionality of the original data and construct a subspace generated by eigenvectors of which each represents the (next) highest variance to explain the data. Let's start at ...
0
votes
1answer
276 views

Autoencoder reconstruction error threshold

I have a set of signals on which I have to implement an anomaly detection algorithm. The data is split among a reference period (i.e. last 3 months) and a test period (i.e. last week). I've already ...
2
votes
3answers
224 views

Simple outlier detection for time series

I wanted to generate a very simple example of anomaly detection for time series. So I created sample data with one very obvious outlier. Here's a picture of the data: The problem is, I didn't get any ...
0
votes
1answer
363 views

Isolation forest with categorical data?

I understand how isolation forests can work with numeric data, but I wonder how it can work with categorical data? Also, at least when working with Sci-kit-Learn, the recommendation I saw was to ...
0
votes
1answer
103 views

What is the difference between sklearn IsolationForests score_samples and decision_function?

The predict method will output -1 (anomaly) where forest.decision_function(X) < forest.threshold_ and 1 otherwise. But what does ...
0
votes
1answer
50 views

Machine Learning alternative for hashing [closed]

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
0
votes
1answer
30 views

Detect anomaly users who try to access too often based on the access log

Does anyone give me advice for statistically detecting anomaly users who try to log in our website too often? At first, the idea that came to my mind is to use Spike detection approaches or IQR ...
3
votes
1answer
169 views

Isolation Forest and average/expected depth formula

The Isolation Forest algorithm (Liu, Fei Tony, Kai Ming Ting, and Zhi-Hua Zhou. "Isolation forest." 2008 Eighth IEEE International Conference on Data Mining. IEEE, 2008. - link: https://cs.nju.edu.cn/...
0
votes
0answers
23 views

how to detect ouliers in skewed data(left skewed distribution) [duplicate]

I have studied a lot of ways of dealing with outliers of normal or multidimensional data. But my problem is about skewed data. How can I find outliers for data with a skewed distribution to the left?
0
votes
0answers
230 views

Interpreting Local Outlier Factor (LOF) results

Using this example on the scikit-learn site, I am attempting to do some anomaly detection using LOF. What I end up with is this: ...
3
votes
2answers
91 views

anomaly detection in time series training data

I have a dataset which basically is measuring the number of people passing a certain region which is monitored and I basically have these raw counts of people over the last two months at 5 minutes ...
0
votes
0answers
23 views

Find plausible peaks in streamed data

i have got a signal of a streamed source which produces values like in the picture. I want to get the "real" peaks (blue circles). But the noisy peaks (red circles) mess up the peak search. The ...
0
votes
0answers
16 views

Detection of a change of a line relative to other lines in time series after some point

I want to detect a change of a line compare to other lines (preferably with statistical significance measure) in a time series after some point. Please, take a look at the plot below (lines are ...
0
votes
0answers
72 views

Unsupervised binary classification with mixed data: Anomaly detection?

I apologize for the lengthy question, but this problem has been troubling me for quite some time now and I can't seem to find an answer to another question which directly applies to my situation. I ...
0
votes
2answers
75 views

Supervised anomaly detection of multiple time series

I'd like to develop a set of models for anomaly detection of multiple time series. After some reading, I have found a few promising approaches, such as Segmentation-based approaches (SECODA); ...
1
vote
1answer
48 views

How detrend a line if i know the slope? [closed]

I am using linear regression to get the slope of some data. If i know the slope how can i flatten the line so that it has no slope?
0
votes
0answers
16 views

Robust way to calculate anomalies of short time series

My question is not related to any actual piece of code, but rather with the thought behind it. I have a short time series of about 20 years of climate data and I want to calculate anomalies. Usually,...
1
vote
2answers
36 views

scikit-learn IsolationForest no variance feature

I'm using IsolationForest algorithm in order to detect anomalies in my data and to use this model to detect future anomalies in new rows and came across a few questions: Is the model good for ...
0
votes
1answer
34 views

Clarification in Anomaly Detection Algorithm

I am referring to Prof Andrew Ng Coursera ML notes (Week 9). He says that to identify outliers we first model the training data and then fit a Gaussian distribution with probability density $ p(X; \mu ...
0
votes
0answers
19 views

Help in Handling multiple classes in independent categorical variables and improve performance

The dataset has 4 categorical and 1 numerical variable and a timestamp variable. Out of 4, three categorical variables are having more than 100 categories. I tried doing one-hot encoding on the whole ...
0
votes
0answers
23 views

Anomaly detection through distribution-based hypothesis testing

I have a table of event logs that contains several categorical variables (gender, age bucket, city of residence, and education level), and I'd like to retroactively identify if a given hour of logs ...
0
votes
0answers
28 views

Setting Choosing alpha for generalized extreme Studentized deviate ESD

I'm working on a S-H-ESD implementation and I'm struggling to set the alpha for the ESD. The suggested alpha everywhere is 0.05. Is there a way to calculate an alpha based on the expected percent of ...

1 2 3 4 5 6