Questions tagged [isolation-forest]
Isolation forest is a variation on random forest used to identify anomalies.
45
questions
0
votes
0
answers
6
views
Figure comparing path length and density (figure 3a) in paper "Isolation-based Anomaly Detection" by Liu, Ting and Zhou
In the paper Isolation-based Anomaly Detection by Liu, Ting and Zhou, figure 3a compares the density and path length for a cluster of anomalies. Figure 3a along with the sub-text has been provided as ...
0
votes
0
answers
7
views
Significance of challenges cited, in creating anomaly scores out of path length, in original paper for Isolation Forest
In the original paper on Isolation Forest by Liu, Ting and Zhou, the authors cite some problems in creating anomaly scores out of the path length $h(x)$ of a point $x$ in an iTree. The text reads as ...
0
votes
0
answers
32
views
Isolation forest subsampling
I have a small dataset which i want to look for anomalies with isolation forest.
In all the papers I read, I see that sub-sampling is an advantage (I'm sure I fully understand why).
Since my data set ...
- 31
0
votes
1
answer
40
views
Detection outliers in financial time series taking into account related time series
I would like seek advice on how to build an efficient approach to identify outliers in a financial series taking into account also related series.
For example, let's assume the there is a very ...
0
votes
0
answers
39
views
Fit and Predicting with Isolation Forest Model
I have time series data that reflects the visitor count from a particular website. So far, I have around a year worth of data and new data comes in everyday. I am trying to predict the anomalies with ...
- 21
2
votes
1
answer
182
views
Do I need stationary time series data for Isolation Forest Model?
I am trying to predict anomalies using an isolation forest model with daily time series data. Do I need to make sure my data is stationary as I have observed weekly seasonality? I read that you need ...
- 21
0
votes
0
answers
60
views
How to use isolation forest to find outliers on one variable while taking into account another variable
Background/Overview:
I am new to python and isolation forest. I am trying to test out different forms of automatic outlier detection to find managers who are rating their direct reports 'oddly'.
I ...
- 1
0
votes
0
answers
51
views
How to properly use isolation forest for data cleaning in a machine learning project?
I am trying to clean my data set (X) from outliers with isolation forest. Then I am going to use the cleaned data set for supervised learning. Should I first split ...
- 1
3
votes
1
answer
364
views
Anomaly detection, LOF vs IsolationForest
I have a training dataset with 140 000 instances with 140 features. Due to the scale of the dataset I'm experiment with letting a model do the anomaly detection and have tried LocalOutlierFactor and ...
- 223
2
votes
1
answer
48
views
Using unsupervised learning anomoly detection to detect fraud?
How can I ensure that the detected class will correspond to fraud rather than another outcome, given that this is an unsupervised learning approach? To my understanding, such algorithms (e.g., ...
- 21
0
votes
1
answer
83
views
Standard ways to automatically remove incorrectly classified observations from a (mostly categorical) training dataset?
I need a model that, given a (mostly categorical) labelled training data, cleans it up, removing incorrectly classified observations. What are the standard techniques to automatically detect and ...
0
votes
1
answer
325
views
How to interpret Isolation Forest results on variations of train/test sets?
I have a labelled dataset, originally intended for classification or clustering tasks, whose minority class is at 10%. I am investigating whether this problem can be tackled with anomaly detection ...
0
votes
1
answer
762
views
Feature Selection in Isolation Forest? How to use kurtosis?
I'm unsure about which is the best approach for feature selection for Isolation Forest. I am using a dataset which initially has 5 numerical columns and 50 categorical. After pre-processing and ...
1
vote
0
answers
210
views
How to detect anomalies in multiple different IP addresses?
Given that my input data consists of various destination IP addresses and its incoming connections from source IP addresses with country codes during certain timestamps, I would like to detect ...
- 119
0
votes
1
answer
207
views
How can I generate a plot of the partitions in Isolation Forests
I have seen this plot is used to indicated how anomalies are isolated via partitioning in Isolation Forests. Is there a library to automatically plot this from a dataset? The plot I want to generate ...
- 101
1
vote
0
answers
911
views
How can we introduce an anomaly class as a positive class to sklearn IsolationForest?
I inspired by this notebook, and I'm experimenting IsolationForest (IF) algorithm using scikit-learn==0.22.2.post1 for anomaly ...
- 409
1
vote
1
answer
1k
views
incorrect results of IsolationForest
I inspired by this notebook, and I'm experimenting IsolationForest algorithm for anomaly detection context on the SF version of KDDCUP99 dataset, including 4 ...
- 409
0
votes
1
answer
980
views
Why is One Class SVM predicting that half my dataset consists of outliers?
I am currently working on a dataset with 14 continuous features, a categorical target over five classes, and 90,000 samples. My current goal is to explore outliers in the dataset, and to that end I ...
1
vote
1
answer
267
views
unsupervised anomaly detection on sparse data
Given that I have a very sparse data matrix with continuous features, like this dataframe for example
...
- 11
2
votes
1
answer
453
views
Isolation Forest - Cost function and optimization method
I have two questions about isolation forest. I may not understand how it works correctly but I just wonder:
What is the cost function of the isolation forest?
What is the optimization method to ...
- 193
1
vote
0
answers
5k
views
How to Tune Isolation Forest?
Many online blogs talk about using Isolation Forest for anomaly detection. But I got a very poor result. The data used is house prices data from Kaggle. I used IForest and KNN from pyod to identify 1% ...
- 887
1
vote
1
answer
3k
views
Anomaly Detection over multivariate categorical and numerical predictors
I am trying to implement Anomaly Detection over a multivariate dataset having categorical and numerical predictors.
If we consider the below sample records, product_type, company_type and currency are ...
- 131
0
votes
0
answers
245
views
Can you use the isolation forest algorithm on large sample sizes?
The original isolation forest paper states that the algorithm works best on small subsamples, but is it okay to use it on large sample sizes or are other anomaly detection algorithms better?
- 111
1
vote
0
answers
315
views
Can you use the isolation forest algorithm on a large sample size?
I've been using the scikit learn sklearn.ensemble.IsolationForest implementation of the isolation forest to detect anomalies in my datasets that range from 100s of ...
- 111
0
votes
0
answers
38
views
Separating anomalys using Isolation Forest, Is my approach correct?
I am a fairly new guy to ML and I am having some trouble choosing a algorithm to the job for me. My data set consists physical measurements where part of the samples were contaminated. In this case, ...
- 1
2
votes
0
answers
61
views
Identify the parameter causing the anomaly in a multivariate dataset
I have a payment transaction dataset with a large number of predictor variables. I am trying to build a model for anomaly detection and I have evaluated various algorithms/approaches for the same like ...
- 131
1
vote
1
answer
726
views
Meaning Of The Terms In Isolation Forest Anomaly Scoring
In an isolation forest the anomaly score of a point is given by:
$$2^{\frac{-E(h(x))}{c(m)}}$$
Now supposedly c(m) is the average length to termination in the search tree. And, E(h(x)) is the ...
- 449
0
votes
1
answer
294
views
Discordance between various methods of multivariate outliers detection
Here is a small "toy example" dataset, with 15 individuals described by 6 variables (this is R language):
...
- 289
0
votes
0
answers
254
views
Oversampling/Undersampling in respect to Train and Test - Isolation Forest
I've got a quite imbalanced data set.
144.496 : 162 -> ratio of 1000:1
I would like to use IsolationForest to detect the 162 anomalys.
I've already split the data.
However, the iForest doesn't ...
- 1
0
votes
0
answers
19
views
Model Examples always with 1 or 2 features
Why are all the model examples that I see on sklearn (e.g., https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.LocalOutlierFactor.html or https://scikit-learn.org/stable/auto_examples/...
- 101
4
votes
1
answer
4k
views
Isolation forest with categorical data?
I understand how isolation forests can work with numeric data, but I wonder how it can work with categorical data?
Also, at least when working with Sci-kit-Learn, the recommendation I saw was to ...
- 3,031
0
votes
0
answers
66
views
Isolation Forest Numerical Example
I'm looking for a proper numerical example to understand Isolation Forests Algorithm correctly. I've read the paper : https://github.com/mgckind/iso_forest/blob/master/icdm08b.pdf, but I want to ...
- 101
6
votes
2
answers
1k
views
Isolation Forest and average/expected depth formula
The Isolation Forest algorithm (Liu, Fei Tony, Kai Ming Ting, and Zhi-Hua Zhou. "Isolation forest." 2008 Eighth IEEE International Conference on Data Mining. IEEE, 2008. - link: https://cs.nju.edu.cn/...
- 547
2
votes
0
answers
509
views
Does the isolation forest care about integer-encoded categorical variables?
The isolation forest (initial paper, follow-up paper) as well as the proposed extended isolation forest (paper) seem like very appealing unsupervised anomaly detection techniques. However, the ...
- 667
5
votes
2
answers
8k
views
How to get top features that contribute to anomalies in Isolation forest
I am using Isolation forest for anomaly detection on multidimensional data. The algorithm is detecting anomalous records with good accuracy. Apart from detecting anomalous records I also need to find ...
- 51
4
votes
2
answers
2k
views
Strange encoding for categorical features
I am reading through https://arxiv.org/pdf/1609.06676.pdf which presents an extension of the isolation forest algorithm so that categorical features may be taken into account. On page 5, the authors ...
- 667
4
votes
2
answers
7k
views
Unsupervised Learning: Train Test division
I have one conceptual question.
In Unsupervised Learning, when I have no labels. The anomaly detection model (Isolation forests, Autoencoders, Distance-based methods etc.), it should fit on a ...
- 145
11
votes
5
answers
15k
views
Feature Importance in Isolation Forest
In an unsupervised setting for higher-dimensional data (e.g. 10 variables (numerical and categorical), 5000 samples, ratio of anomalies likely 1% or below but unknown) I am able to fit the isolation ...
- 667
4
votes
1
answer
2k
views
Feature Importances for Isolation Forest?
Is there a way to dig in to sklearn's Isolation Forest algorithm to understand why data is scored as an inlier or outlier? I see that you can produce a score with decision_function (for data used to ...
- 41
4
votes
0
answers
3k
views
Unsupervised anomaly detection - metric for tuning Isolation Forest parameters
I have a project, in which, one of the stages is to find and label anomalous data points, that are likely to be outliers.
As a first step, I am using Isolation Forest algorithm, which, after plotting ...
- 41
4
votes
1
answer
3k
views
Validating Isolation Forests
I have a dataset where I'd like to perform anomaly detection with an Isolation Forest. I don't have any way to validate the model (my data is not labeled - that's why I'm using unsupervised learning) -...
- 207
14
votes
3
answers
14k
views
scikit-learn IsolationForest anomaly score
According to IsolationForest papers (refs are given in documentation)
the score produced by Isolation Forest should be between 0 and 1.
The implementation in scikit-learn negates the scores (so high ...
- 177
4
votes
0
answers
816
views
What data assumptions are made when using an isolation forest?
I am performing anomaly detection with a high-dimensional zero-inflated data-set. This rules out options like standard Gaussian anomaly detection as no amount of transformation can make my features "...
- 153
1
vote
1
answer
6k
views
Does Isolation Forest need an anomaly sample during training?
I am using Isolation Forest for anomaly detection (scikit implementation in python).
My data have 1000 dimensions. My normal data, which I use for training Isolation Forest model, has only to features ...
- 113
0
votes
0
answers
418
views
Outlier Detection giving almost all training points as outliers
I am trying to run an outlier detection algorithm using python and I have used OneClassSVM and Isolation Forest on a dataset of about 30 points with 5 outliers. I have trained on the 25 points and ...
- 577