Questions tagged [isolation-forest]

Isolation forest is a variation on random forest used to identify anomalies.

Filter by
Sorted by
Tagged with
0 votes
0 answers
30 views

Height limits relation to anomaly score in Isolation Forest

I am trying to implement the Isolation Forest algorithm in Python and faced an issue when dealing with the max_depth and the height limit (l) from the white paper. (See the 2. set height limit l) ...
user avatar
0 votes
1 answer
28 views

Standard ways to automatically remove incorrectly classified observations from a (mostly categorical) training dataset?

I need a model that, given a (mostly categorical) labelled training data, cleans it up, removing incorrectly classified observations. What are the standard techniques to automatically detect and ...
user avatar
0 votes
1 answer
68 views

How to interpret Isolation Forest results on variations of train/test sets?

I have a labelled dataset, originally intended for classification or clustering tasks, whose minority class is at 10%. I am investigating whether this problem can be tackled with anomaly detection ...
user avatar
0 votes
1 answer
349 views

Feature Selection in Isolation Forest? How to use kurtosis?

I'm unsure about which is the best approach for feature selection for Isolation Forest. I am using a dataset which initially has 5 numerical columns and 50 categorical. After pre-processing and ...
user avatar
0 votes
0 answers
70 views

Isolation forest for real time anomaly detection

I currently have isolation forest working for a productivity dataset.But i want to make isolation forestto detect anmomalies in real time manner. pls find below my current code in python. I am ...
user avatar
1 vote
0 answers
66 views

How to detect anomalies in multiple different IP addresses?

Given that my input data consists of various destination IP addresses and its incoming connections from source IP addresses with country codes during certain timestamps, I would like to detect ...
user avatar
  • 119
0 votes
1 answer
62 views

How can I generate a plot of the partitions in Isolation Forests

I have seen this plot is used to indicated how anomalies are isolated via partitioning in Isolation Forests. Is there a library to automatically plot this from a dataset? The plot I want to generate ...
user avatar
  • 101
1 vote
0 answers
551 views

How can we introduce an anomaly class as a positive class to sklearn IsolationForest?

I inspired by this notebook, and I'm experimenting IsolationForest (IF) algorithm using scikit-learn==0.22.2.post1 for anomaly ...
user avatar
  • 385
1 vote
1 answer
583 views

incorrect results of IsolationForest

I inspired by this notebook, and I'm experimenting IsolationForest algorithm for anomaly detection context on the SF version of KDDCUP99 dataset, including 4 ...
user avatar
  • 385
0 votes
1 answer
535 views

Why is One Class SVM predicting that half my dataset consists of outliers?

I am currently working on a dataset with 14 continuous features, a categorical target over five classes, and 90,000 samples. My current goal is to explore outliers in the dataset, and to that end I ...
user avatar
1 vote
1 answer
209 views

unsupervised anomaly detection on sparse data

Given that I have a very sparse data matrix with continuous features, like this dataframe for example ...
user avatar
2 votes
1 answer
261 views

Isolation Forest - Cost function and optimization method

I have two questions about isolation forest. I may not understand how it works correctly but I just wonder: What is the cost function of the isolation forest? What is the optimization method to ...
user avatar
  • 173
0 votes
0 answers
3k views

How to Tune Isolation Forest?

Many online blogs talk about using Isolation Forest for anomaly detection. But I got a very poor result. The data used is house prices data from Kaggle. I used IForest and KNN from pyod to identify 1% ...
user avatar
  • 643
1 vote
1 answer
2k views

Anomaly Detection over multivariate categorical and numerical predictors

I am trying to implement Anomaly Detection over a multivariate dataset having categorical and numerical predictors. If we consider the below sample records, product_type, company_type and currency are ...
user avatar
0 votes
0 answers
169 views

Can you use the isolation forest algorithm on large sample sizes?

The original isolation forest paper states that the algorithm works best on small subsamples, but is it okay to use it on large sample sizes or are other anomaly detection algorithms better?
user avatar
  • 111
1 vote
0 answers
201 views

Can you use the isolation forest algorithm on a large sample size?

I've been using the scikit learn sklearn.ensemble.IsolationForest implementation of the isolation forest to detect anomalies in my datasets that range from 100s of ...
user avatar
  • 111
0 votes
0 answers
30 views

Separating anomalys using Isolation Forest, Is my approach correct?

I am a fairly new guy to ML and I am having some trouble choosing a algorithm to the job for me. My data set consists physical measurements where part of the samples were contaminated. In this case, ...
user avatar
2 votes
0 answers
54 views

Identify the parameter causing the anomaly in a multivariate dataset

I have a payment transaction dataset with a large number of predictor variables. I am trying to build a model for anomaly detection and I have evaluated various algorithms/approaches for the same like ...
user avatar
1 vote
1 answer
412 views

Meaning Of The Terms In Isolation Forest Anomaly Scoring

In an isolation forest the anomaly score of a point is given by: $$2^{\frac{-E(h(x))}{c(m)}}$$ Now supposedly c(m) is the average length to termination in the search tree. And, E(h(x)) is the ...
user avatar
  • 379
0 votes
1 answer
160 views

Discordance between various methods of multivariate outliers detection

Here is a small "toy example" dataset, with 15 individuals described by 6 variables (this is R language): ...
user avatar
0 votes
0 answers
191 views

Oversampling/Undersampling in respect to Train and Test - Isolation Forest

I've got a quite imbalanced data set. 144.496 : 162 -> ratio of 1000:1 I would like to use IsolationForest to detect the 162 anomalys. I've already split the data. However, the iForest doesn't ...
user avatar
  • 1
0 votes
0 answers
19 views

Model Examples always with 1 or 2 features

Why are all the model examples that I see on sklearn (e.g., https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.LocalOutlierFactor.html or https://scikit-learn.org/stable/auto_examples/...
user avatar
  • 101
2 votes
1 answer
2k views

Isolation forest with categorical data?

I understand how isolation forests can work with numeric data, but I wonder how it can work with categorical data? Also, at least when working with Sci-kit-Learn, the recommendation I saw was to ...
user avatar
0 votes
0 answers
60 views

Isolation Forest Numerical Example

I'm looking for a proper numerical example to understand Isolation Forests Algorithm correctly. I've read the paper : https://github.com/mgckind/iso_forest/blob/master/icdm08b.pdf, but I want to ...
user avatar
  • 101
6 votes
2 answers
844 views

Isolation Forest and average/expected depth formula

The Isolation Forest algorithm (Liu, Fei Tony, Kai Ming Ting, and Zhi-Hua Zhou. "Isolation forest." 2008 Eighth IEEE International Conference on Data Mining. IEEE, 2008. - link: https://cs.nju.edu.cn/...
user avatar
2 votes
0 answers
398 views

Does the isolation forest care about integer-encoded categorical variables?

The isolation forest (initial paper, follow-up paper) as well as the proposed extended isolation forest (paper) seem like very appealing unsupervised anomaly detection techniques. However, the ...
user avatar
5 votes
2 answers
6k views

How to get top features that contribute to anomalies in Isolation forest

I am using Isolation forest for anomaly detection on multidimensional data. The algorithm is detecting anomalous records with good accuracy. Apart from detecting anomalous records I also need to find ...
user avatar
  • 51
4 votes
2 answers
1k views

Strange encoding for categorical features

I am reading through https://arxiv.org/pdf/1609.06676.pdf which presents an extension of the isolation forest algorithm so that categorical features may be taken into account. On page 5, the authors ...
user avatar
4 votes
2 answers
5k views

Unsupervised Learning: Train Test division

I have one conceptual question. In Unsupervised Learning, when I have no labels. The anomaly detection model (Isolation forests, Autoencoders, Distance-based methods etc.), it should fit on a ...
user avatar
11 votes
4 answers
10k views

Feature Importance in Isolation Forest

In an unsupervised setting for higher-dimensional data (e.g. 10 variables (numerical and categorical), 5000 samples, ratio of anomalies likely 1% or below but unknown) I am able to fit the isolation ...
user avatar
2 votes
0 answers
1k views

Feature Importances for Isolation Forest?

Is there a way to dig in to sklearn's Isolation Forest algorithm to understand why data is scored as an inlier or outlier? I see that you can produce a score with decision_function (for data used to ...
user avatar
  • 21
3 votes
0 answers
2k views

Unsupervised anomaly detection - metric for tuning Isolation Forest parameters

I have a project, in which, one of the stages is to find and label anomalous data points, that are likely to be outliers. As a first step, I am using Isolation Forest algorithm, which, after plotting ...
user avatar
4 votes
1 answer
2k views

Validating Isolation Forests

I have a dataset where I'd like to perform anomaly detection with an Isolation Forest. I don't have any way to validate the model (my data is not labeled - that's why I'm using unsupervised learning) -...
user avatar
  • 207
13 votes
3 answers
11k views

scikit-learn IsolationForest anomaly score

According to IsolationForest papers (refs are given in documentation) the score produced by Isolation Forest should be between 0 and 1. The implementation in scikit-learn negates the scores (so high ...
user avatar
  • 167
4 votes
0 answers
711 views

What data assumptions are made when using an isolation forest?

I am performing anomaly detection with a high-dimensional zero-inflated data-set. This rules out options like standard Gaussian anomaly detection as no amount of transformation can make my features "...
user avatar
1 vote
1 answer
6k views

Does Isolation Forest need an anomaly sample during training?

I am using Isolation Forest for anomaly detection (scikit implementation in python). My data have 1000 dimensions. My normal data, which I use for training Isolation Forest model, has only to features ...
user avatar
0 votes
0 answers
414 views

Outlier Detection giving almost all training points as outliers

I am trying to run an outlier detection algorithm using python and I have used OneClassSVM and Isolation Forest on a dataset of about 30 points with 5 outliers. I have trained on the 25 points and ...
user avatar