Questions tagged [isolation-forest]

Isolation forest is a variation on random forest used to identify anomalies.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

Figure comparing path length and density (figure 3a) in paper "Isolation-based Anomaly Detection" by Liu, Ting and Zhou

In the paper Isolation-based Anomaly Detection by Liu, Ting and Zhou, figure 3a compares the density and path length for a cluster of anomalies. Figure 3a along with the sub-text has been provided as ...
Anirban Chakraborty's user avatar
0 votes
0 answers
7 views

Significance of challenges cited, in creating anomaly scores out of path length, in original paper for Isolation Forest

In the original paper on Isolation Forest by Liu, Ting and Zhou, the authors cite some problems in creating anomaly scores out of the path length $h(x)$ of a point $x$ in an iTree. The text reads as ...
Anirban Chakraborty's user avatar
0 votes
0 answers
32 views

Isolation forest subsampling

I have a small dataset which i want to look for anomalies with isolation forest. In all the papers I read, I see that sub-sampling is an advantage (I'm sure I fully understand why). Since my data set ...
user2519685's user avatar
0 votes
1 answer
40 views

Detection outliers in financial time series taking into account related time series

I would like seek advice on how to build an efficient approach to identify outliers in a financial series taking into account also related series. For example, let's assume the there is a very ...
user3548751's user avatar
0 votes
0 answers
39 views

Fit and Predicting with Isolation Forest Model

I have time series data that reflects the visitor count from a particular website. So far, I have around a year worth of data and new data comes in everyday. I am trying to predict the anomalies with ...
Henry's user avatar
  • 21
2 votes
1 answer
182 views

Do I need stationary time series data for Isolation Forest Model?

I am trying to predict anomalies using an isolation forest model with daily time series data. Do I need to make sure my data is stationary as I have observed weekly seasonality? I read that you need ...
Henry's user avatar
  • 21
0 votes
0 answers
60 views

How to use isolation forest to find outliers on one variable while taking into account another variable

Background/Overview: I am new to python and isolation forest. I am trying to test out different forms of automatic outlier detection to find managers who are rating their direct reports 'oddly'. I ...
Silvok1's user avatar
0 votes
0 answers
51 views

How to properly use isolation forest for data cleaning in a machine learning project?

I am trying to clean my data set (X) from outliers with isolation forest. Then I am going to use the cleaned data set for supervised learning. Should I first split ...
Hesan's user avatar
  • 1
3 votes
1 answer
364 views

Anomaly detection, LOF vs IsolationForest

I have a training dataset with 140 000 instances with 140 features. Due to the scale of the dataset I'm experiment with letting a model do the anomaly detection and have tried LocalOutlierFactor and ...
Henri's user avatar
  • 223
2 votes
1 answer
48 views

Using unsupervised learning anomoly detection to detect fraud?

How can I ensure that the detected class will correspond to fraud rather than another outcome, given that this is an unsupervised learning approach? To my understanding, such algorithms (e.g., ...
user362213's user avatar
0 votes
1 answer
83 views

Standard ways to automatically remove incorrectly classified observations from a (mostly categorical) training dataset?

I need a model that, given a (mostly categorical) labelled training data, cleans it up, removing incorrectly classified observations. What are the standard techniques to automatically detect and ...
Giuliano Mirabella's user avatar
0 votes
1 answer
325 views

How to interpret Isolation Forest results on variations of train/test sets?

I have a labelled dataset, originally intended for classification or clustering tasks, whose minority class is at 10%. I am investigating whether this problem can be tackled with anomaly detection ...
idontknowmuch's user avatar
0 votes
1 answer
762 views

Feature Selection in Isolation Forest? How to use kurtosis?

I'm unsure about which is the best approach for feature selection for Isolation Forest. I am using a dataset which initially has 5 numerical columns and 50 categorical. After pre-processing and ...
idontknowmuch's user avatar
1 vote
0 answers
210 views

How to detect anomalies in multiple different IP addresses?

Given that my input data consists of various destination IP addresses and its incoming connections from source IP addresses with country codes during certain timestamps, I would like to detect ...
Viktoria's user avatar
  • 119
0 votes
1 answer
207 views

How can I generate a plot of the partitions in Isolation Forests

I have seen this plot is used to indicated how anomalies are isolated via partitioning in Isolation Forests. Is there a library to automatically plot this from a dataset? The plot I want to generate ...
theSekyi's user avatar
  • 101
1 vote
0 answers
911 views

How can we introduce an anomaly class as a positive class to sklearn IsolationForest?

I inspired by this notebook, and I'm experimenting IsolationForest (IF) algorithm using scikit-learn==0.22.2.post1 for anomaly ...
Mario's user avatar
  • 409
1 vote
1 answer
1k views

incorrect results of IsolationForest

I inspired by this notebook, and I'm experimenting IsolationForest algorithm for anomaly detection context on the SF version of KDDCUP99 dataset, including 4 ...
Mario's user avatar
  • 409
0 votes
1 answer
980 views

Why is One Class SVM predicting that half my dataset consists of outliers?

I am currently working on a dataset with 14 continuous features, a categorical target over five classes, and 90,000 samples. My current goal is to explore outliers in the dataset, and to that end I ...
Tarek Allam's user avatar
1 vote
1 answer
267 views

unsupervised anomaly detection on sparse data

Given that I have a very sparse data matrix with continuous features, like this dataframe for example ...
greghouse1's user avatar
2 votes
1 answer
453 views

Isolation Forest - Cost function and optimization method

I have two questions about isolation forest. I may not understand how it works correctly but I just wonder: What is the cost function of the isolation forest? What is the optimization method to ...
Amhs_11's user avatar
  • 193
1 vote
0 answers
5k views

How to Tune Isolation Forest?

Many online blogs talk about using Isolation Forest for anomaly detection. But I got a very poor result. The data used is house prices data from Kaggle. I used IForest and KNN from pyod to identify 1% ...
etang's user avatar
  • 887
1 vote
1 answer
3k views

Anomaly Detection over multivariate categorical and numerical predictors

I am trying to implement Anomaly Detection over a multivariate dataset having categorical and numerical predictors. If we consider the below sample records, product_type, company_type and currency are ...
Dhaval Simaria's user avatar
0 votes
0 answers
245 views

Can you use the isolation forest algorithm on large sample sizes?

The original isolation forest paper states that the algorithm works best on small subsamples, but is it okay to use it on large sample sizes or are other anomaly detection algorithms better?
ddx's user avatar
  • 111
1 vote
0 answers
315 views

Can you use the isolation forest algorithm on a large sample size?

I've been using the scikit learn sklearn.ensemble.IsolationForest implementation of the isolation forest to detect anomalies in my datasets that range from 100s of ...
ddx's user avatar
  • 111
0 votes
0 answers
38 views

Separating anomalys using Isolation Forest, Is my approach correct?

I am a fairly new guy to ML and I am having some trouble choosing a algorithm to the job for me. My data set consists physical measurements where part of the samples were contaminated. In this case, ...
Back up's user avatar
2 votes
0 answers
61 views

Identify the parameter causing the anomaly in a multivariate dataset

I have a payment transaction dataset with a large number of predictor variables. I am trying to build a model for anomaly detection and I have evaluated various algorithms/approaches for the same like ...
Dhaval Simaria's user avatar
1 vote
1 answer
726 views

Meaning Of The Terms In Isolation Forest Anomaly Scoring

In an isolation forest the anomaly score of a point is given by: $$2^{\frac{-E(h(x))}{c(m)}}$$ Now supposedly c(m) is the average length to termination in the search tree. And, E(h(x)) is the ...
Trajan's user avatar
  • 449
0 votes
1 answer
294 views

Discordance between various methods of multivariate outliers detection

Here is a small "toy example" dataset, with 15 individuals described by 6 variables (this is R language): ...
Philopolis's user avatar
0 votes
0 answers
254 views

Oversampling/Undersampling in respect to Train and Test - Isolation Forest

I've got a quite imbalanced data set. 144.496 : 162 -> ratio of 1000:1 I would like to use IsolationForest to detect the 162 anomalys. I've already split the data. However, the iForest doesn't ...
Marvin's user avatar
  • 1
0 votes
0 answers
19 views

Model Examples always with 1 or 2 features

Why are all the model examples that I see on sklearn (e.g., https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.LocalOutlierFactor.html or https://scikit-learn.org/stable/auto_examples/...
theStud54's user avatar
  • 101
4 votes
1 answer
4k views

Isolation forest with categorical data?

I understand how isolation forests can work with numeric data, but I wonder how it can work with categorical data? Also, at least when working with Sci-kit-Learn, the recommendation I saw was to ...
Maverick Meerkat's user avatar
0 votes
0 answers
66 views

Isolation Forest Numerical Example

I'm looking for a proper numerical example to understand Isolation Forests Algorithm correctly. I've read the paper : https://github.com/mgckind/iso_forest/blob/master/icdm08b.pdf, but I want to ...
zubug55's user avatar
  • 101
6 votes
2 answers
1k views

Isolation Forest and average/expected depth formula

The Isolation Forest algorithm (Liu, Fei Tony, Kai Ming Ting, and Zhi-Hua Zhou. "Isolation forest." 2008 Eighth IEEE International Conference on Data Mining. IEEE, 2008. - link: https://cs.nju.edu.cn/...
anymous.asker's user avatar
2 votes
0 answers
509 views

Does the isolation forest care about integer-encoded categorical variables?

The isolation forest (initial paper, follow-up paper) as well as the proposed extended isolation forest (paper) seem like very appealing unsupervised anomaly detection techniques. However, the ...
robot_2077198's user avatar
5 votes
2 answers
8k views

How to get top features that contribute to anomalies in Isolation forest

I am using Isolation forest for anomaly detection on multidimensional data. The algorithm is detecting anomalous records with good accuracy. Apart from detecting anomalous records I also need to find ...
Amar's user avatar
  • 51
4 votes
2 answers
2k views

Strange encoding for categorical features

I am reading through https://arxiv.org/pdf/1609.06676.pdf which presents an extension of the isolation forest algorithm so that categorical features may be taken into account. On page 5, the authors ...
robot_2077198's user avatar
4 votes
2 answers
7k views

Unsupervised Learning: Train Test division

I have one conceptual question. In Unsupervised Learning, when I have no labels. The anomaly detection model (Isolation forests, Autoencoders, Distance-based methods etc.), it should fit on a ...
Manu Sharma's user avatar
11 votes
5 answers
15k views

Feature Importance in Isolation Forest

In an unsupervised setting for higher-dimensional data (e.g. 10 variables (numerical and categorical), 5000 samples, ratio of anomalies likely 1% or below but unknown) I am able to fit the isolation ...
robot_2077198's user avatar
4 votes
1 answer
2k views

Feature Importances for Isolation Forest?

Is there a way to dig in to sklearn's Isolation Forest algorithm to understand why data is scored as an inlier or outlier? I see that you can produce a score with decision_function (for data used to ...
Maripqz's user avatar
  • 41
4 votes
0 answers
3k views

Unsupervised anomaly detection - metric for tuning Isolation Forest parameters

I have a project, in which, one of the stages is to find and label anomalous data points, that are likely to be outliers. As a first step, I am using Isolation Forest algorithm, which, after plotting ...
Bonzogondo's user avatar
4 votes
1 answer
3k views

Validating Isolation Forests

I have a dataset where I'd like to perform anomaly detection with an Isolation Forest. I don't have any way to validate the model (my data is not labeled - that's why I'm using unsupervised learning) -...
lte__'s user avatar
  • 207
14 votes
3 answers
14k views

scikit-learn IsolationForest anomaly score

According to IsolationForest papers (refs are given in documentation) the score produced by Isolation Forest should be between 0 and 1. The implementation in scikit-learn negates the scores (so high ...
DAF's user avatar
  • 177
4 votes
0 answers
816 views

What data assumptions are made when using an isolation forest?

I am performing anomaly detection with a high-dimensional zero-inflated data-set. This rules out options like standard Gaussian anomaly detection as no amount of transformation can make my features "...
Darrrrrren's user avatar
1 vote
1 answer
6k views

Does Isolation Forest need an anomaly sample during training?

I am using Isolation Forest for anomaly detection (scikit implementation in python). My data have 1000 dimensions. My normal data, which I use for training Isolation Forest model, has only to features ...
user1872329's user avatar
0 votes
0 answers
418 views

Outlier Detection giving almost all training points as outliers

I am trying to run an outlier detection algorithm using python and I have used OneClassSVM and Isolation Forest on a dataset of about 30 points with 5 outliers. I have trained on the 25 points and ...
silent_dev's user avatar