Questions tagged [isolation-forest]
Isolation forest is a variation on random forest used to identify anomalies.
37
questions
0
votes
0
answers
30
views
Height limits relation to anomaly score in Isolation Forest
I am trying to implement the Isolation Forest algorithm in Python and faced an issue when dealing with the max_depth and the height limit (l) from the white paper. (See the 2. set height limit l)
...
- 1
0
votes
1
answer
28
views
Standard ways to automatically remove incorrectly classified observations from a (mostly categorical) training dataset?
I need a model that, given a (mostly categorical) labelled training data, cleans it up, removing incorrectly classified observations. What are the standard techniques to automatically detect and ...
0
votes
1
answer
68
views
How to interpret Isolation Forest results on variations of train/test sets?
I have a labelled dataset, originally intended for classification or clustering tasks, whose minority class is at 10%. I am investigating whether this problem can be tackled with anomaly detection ...
0
votes
1
answer
349
views
Feature Selection in Isolation Forest? How to use kurtosis?
I'm unsure about which is the best approach for feature selection for Isolation Forest. I am using a dataset which initially has 5 numerical columns and 50 categorical. After pre-processing and ...
0
votes
0
answers
70
views
Isolation forest for real time anomaly detection
I currently have isolation forest working for a productivity dataset.But i want to make isolation forestto detect anmomalies in real time manner. pls find below my current code in python. I am ...
1
vote
0
answers
66
views
How to detect anomalies in multiple different IP addresses?
Given that my input data consists of various destination IP addresses and its incoming connections from source IP addresses with country codes during certain timestamps, I would like to detect ...
- 119
0
votes
1
answer
62
views
How can I generate a plot of the partitions in Isolation Forests
I have seen this plot is used to indicated how anomalies are isolated via partitioning in Isolation Forests. Is there a library to automatically plot this from a dataset? The plot I want to generate ...
- 101
1
vote
0
answers
551
views
How can we introduce an anomaly class as a positive class to sklearn IsolationForest?
I inspired by this notebook, and I'm experimenting IsolationForest (IF) algorithm using scikit-learn==0.22.2.post1 for anomaly ...
- 385
1
vote
1
answer
583
views
incorrect results of IsolationForest
I inspired by this notebook, and I'm experimenting IsolationForest algorithm for anomaly detection context on the SF version of KDDCUP99 dataset, including 4 ...
- 385
0
votes
1
answer
535
views
Why is One Class SVM predicting that half my dataset consists of outliers?
I am currently working on a dataset with 14 continuous features, a categorical target over five classes, and 90,000 samples. My current goal is to explore outliers in the dataset, and to that end I ...
1
vote
1
answer
209
views
unsupervised anomaly detection on sparse data
Given that I have a very sparse data matrix with continuous features, like this dataframe for example
...
- 11
2
votes
1
answer
261
views
Isolation Forest - Cost function and optimization method
I have two questions about isolation forest. I may not understand how it works correctly but I just wonder:
What is the cost function of the isolation forest?
What is the optimization method to ...
- 173
0
votes
0
answers
3k
views
How to Tune Isolation Forest?
Many online blogs talk about using Isolation Forest for anomaly detection. But I got a very poor result. The data used is house prices data from Kaggle. I used IForest and KNN from pyod to identify 1% ...
- 643
1
vote
1
answer
2k
views
Anomaly Detection over multivariate categorical and numerical predictors
I am trying to implement Anomaly Detection over a multivariate dataset having categorical and numerical predictors.
If we consider the below sample records, product_type, company_type and currency are ...
- 131
0
votes
0
answers
169
views
Can you use the isolation forest algorithm on large sample sizes?
The original isolation forest paper states that the algorithm works best on small subsamples, but is it okay to use it on large sample sizes or are other anomaly detection algorithms better?
- 111
1
vote
0
answers
201
views
Can you use the isolation forest algorithm on a large sample size?
I've been using the scikit learn sklearn.ensemble.IsolationForest implementation of the isolation forest to detect anomalies in my datasets that range from 100s of ...
- 111
0
votes
0
answers
30
views
Separating anomalys using Isolation Forest, Is my approach correct?
I am a fairly new guy to ML and I am having some trouble choosing a algorithm to the job for me. My data set consists physical measurements where part of the samples were contaminated. In this case, ...
- 1
2
votes
0
answers
54
views
Identify the parameter causing the anomaly in a multivariate dataset
I have a payment transaction dataset with a large number of predictor variables. I am trying to build a model for anomaly detection and I have evaluated various algorithms/approaches for the same like ...
- 131
1
vote
1
answer
412
views
Meaning Of The Terms In Isolation Forest Anomaly Scoring
In an isolation forest the anomaly score of a point is given by:
$$2^{\frac{-E(h(x))}{c(m)}}$$
Now supposedly c(m) is the average length to termination in the search tree. And, E(h(x)) is the ...
- 379
0
votes
1
answer
160
views
Discordance between various methods of multivariate outliers detection
Here is a small "toy example" dataset, with 15 individuals described by 6 variables (this is R language):
...
- 289
0
votes
0
answers
191
views
Oversampling/Undersampling in respect to Train and Test - Isolation Forest
I've got a quite imbalanced data set.
144.496 : 162 -> ratio of 1000:1
I would like to use IsolationForest to detect the 162 anomalys.
I've already split the data.
However, the iForest doesn't ...
- 1
0
votes
0
answers
19
views
Model Examples always with 1 or 2 features
Why are all the model examples that I see on sklearn (e.g., https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.LocalOutlierFactor.html or https://scikit-learn.org/stable/auto_examples/...
- 101
2
votes
1
answer
2k
views
Isolation forest with categorical data?
I understand how isolation forests can work with numeric data, but I wonder how it can work with categorical data?
Also, at least when working with Sci-kit-Learn, the recommendation I saw was to ...
- 2,217
0
votes
0
answers
60
views
Isolation Forest Numerical Example
I'm looking for a proper numerical example to understand Isolation Forests Algorithm correctly. I've read the paper : https://github.com/mgckind/iso_forest/blob/master/icdm08b.pdf, but I want to ...
- 101
6
votes
2
answers
844
views
Isolation Forest and average/expected depth formula
The Isolation Forest algorithm (Liu, Fei Tony, Kai Ming Ting, and Zhi-Hua Zhou. "Isolation forest." 2008 Eighth IEEE International Conference on Data Mining. IEEE, 2008. - link: https://cs.nju.edu.cn/...
- 525
2
votes
0
answers
398
views
Does the isolation forest care about integer-encoded categorical variables?
The isolation forest (initial paper, follow-up paper) as well as the proposed extended isolation forest (paper) seem like very appealing unsupervised anomaly detection techniques. However, the ...
- 607
5
votes
2
answers
6k
views
How to get top features that contribute to anomalies in Isolation forest
I am using Isolation forest for anomaly detection on multidimensional data. The algorithm is detecting anomalous records with good accuracy. Apart from detecting anomalous records I also need to find ...
- 51
4
votes
2
answers
1k
views
Strange encoding for categorical features
I am reading through https://arxiv.org/pdf/1609.06676.pdf which presents an extension of the isolation forest algorithm so that categorical features may be taken into account. On page 5, the authors ...
- 607
4
votes
2
answers
5k
views
Unsupervised Learning: Train Test division
I have one conceptual question.
In Unsupervised Learning, when I have no labels. The anomaly detection model (Isolation forests, Autoencoders, Distance-based methods etc.), it should fit on a ...
- 145
11
votes
4
answers
10k
views
Feature Importance in Isolation Forest
In an unsupervised setting for higher-dimensional data (e.g. 10 variables (numerical and categorical), 5000 samples, ratio of anomalies likely 1% or below but unknown) I am able to fit the isolation ...
- 607
2
votes
0
answers
1k
views
Feature Importances for Isolation Forest?
Is there a way to dig in to sklearn's Isolation Forest algorithm to understand why data is scored as an inlier or outlier? I see that you can produce a score with decision_function (for data used to ...
- 21
3
votes
0
answers
2k
views
Unsupervised anomaly detection - metric for tuning Isolation Forest parameters
I have a project, in which, one of the stages is to find and label anomalous data points, that are likely to be outliers.
As a first step, I am using Isolation Forest algorithm, which, after plotting ...
- 31
4
votes
1
answer
2k
views
Validating Isolation Forests
I have a dataset where I'd like to perform anomaly detection with an Isolation Forest. I don't have any way to validate the model (my data is not labeled - that's why I'm using unsupervised learning) -...
- 207
13
votes
3
answers
11k
views
scikit-learn IsolationForest anomaly score
According to IsolationForest papers (refs are given in documentation)
the score produced by Isolation Forest should be between 0 and 1.
The implementation in scikit-learn negates the scores (so high ...
- 167
4
votes
0
answers
711
views
What data assumptions are made when using an isolation forest?
I am performing anomaly detection with a high-dimensional zero-inflated data-set. This rules out options like standard Gaussian anomaly detection as no amount of transformation can make my features "...
- 153
1
vote
1
answer
6k
views
Does Isolation Forest need an anomaly sample during training?
I am using Isolation Forest for anomaly detection (scikit implementation in python).
My data have 1000 dimensions. My normal data, which I use for training Isolation Forest model, has only to features ...
- 113
0
votes
0
answers
414
views
Outlier Detection giving almost all training points as outliers
I am trying to run an outlier detection algorithm using python and I have used OneClassSVM and Isolation Forest on a dataset of about 30 points with 5 outliers. I have trained on the 25 points and ...
- 567