Skip to main content

Questions tagged [isolation-forest]

Isolation forest is a variation on random forest used to identify anomalies.

Filter by
Sorted by
Tagged with
2 votes
0 answers
20 views

Time Series Anomaly Detection with Class Variables

In ("univariate") time series anomaly detection, what techniques are there to incorporate class variables? For example, in accounting transactional data, we might care about anomalies in the ...
olives's user avatar
  • 31
1 vote
0 answers
18 views

Does it make sense to increase the subsampling size for Isolation Forest in the context of sparse matrix data?

I am currently working with a sparse matrix dataset and employing the Isolation Forest algorithm for outlier detection. Given the nature of my data (high dimensionality and sparsity), I am ...
Camilo Piñón's user avatar
0 votes
1 answer
50 views

Detection outliers in financial time series taking into account related time series

I would like seek advice on how to build an efficient approach to identify outliers in a financial series taking into account also related series. For example, let's assume the there is a very ...
user3548751's user avatar
4 votes
1 answer
513 views

Do I need stationary time series data for Isolation Forest Model?

I am trying to predict anomalies using an isolation forest model with daily time series data. Do I need to make sure my data is stationary as I have observed weekly seasonality? I read that you need ...
Henry's user avatar
  • 41
4 votes
1 answer
748 views

Anomaly detection, LOF vs IsolationForest

I have a training dataset with 140 000 instances with 140 features. Due to the scale of the dataset I'm experiment with letting a model do the anomaly detection and have tried LocalOutlierFactor and ...
Henri's user avatar
  • 243
2 votes
1 answer
63 views

Using unsupervised learning anomoly detection to detect fraud?

How can I ensure that the detected class will correspond to fraud rather than another outcome, given that this is an unsupervised learning approach? To my understanding, such algorithms (e.g., ...
user362213's user avatar
0 votes
1 answer
112 views

Standard ways to automatically remove incorrectly classified observations from a (mostly categorical) training dataset?

I need a model that, given a (mostly categorical) labelled training data, cleans it up, removing incorrectly classified observations. What are the standard techniques to automatically detect and ...
Giuliano Mirabella's user avatar
0 votes
1 answer
546 views

How to interpret Isolation Forest results on variations of train/test sets?

I have a labelled dataset, originally intended for classification or clustering tasks, whose minority class is at 10%. I am investigating whether this problem can be tackled with anomaly detection ...
idontknowmuch's user avatar
1 vote
1 answer
1k views

Feature Selection in Isolation Forest? How to use kurtosis?

I'm unsure about which is the best approach for feature selection for Isolation Forest. I am using a dataset which initially has 5 numerical columns and 50 categorical. After pre-processing and ...
idontknowmuch's user avatar
1 vote
0 answers
383 views

How to detect anomalies in multiple different IP addresses?

Given that my input data consists of various destination IP addresses and its incoming connections from source IP addresses with country codes during certain timestamps, I would like to detect ...
Viktoria's user avatar
  • 119
0 votes
1 answer
288 views

How can I generate a plot of the partitions in Isolation Forests

I have seen this plot is used to indicated how anomalies are isolated via partitioning in Isolation Forests. Is there a library to automatically plot this from a dataset? The plot I want to generate ...
theSekyi's user avatar
  • 101
1 vote
0 answers
1k views

How can we introduce an anomaly class as a positive class to sklearn IsolationForest?

I inspired by this notebook, and I'm experimenting IsolationForest (IF) algorithm using scikit-learn==0.22.2.post1 for anomaly ...
Mario's user avatar
  • 441
1 vote
1 answer
2k views

incorrect results of IsolationForest

I inspired by this notebook, and I'm experimenting IsolationForest algorithm for anomaly detection context on the SF version of KDDCUP99 dataset, including 4 ...
Mario's user avatar
  • 441
0 votes
1 answer
1k views

Why is One Class SVM predicting that half my dataset consists of outliers?

I am currently working on a dataset with 14 continuous features, a categorical target over five classes, and 90,000 samples. My current goal is to explore outliers in the dataset, and to that end I ...
Tarek Allam's user avatar
1 vote
1 answer
313 views

unsupervised anomaly detection on sparse data

Given that I have a very sparse data matrix with continuous features, like this dataframe for example ...
greghouse1's user avatar
2 votes
1 answer
567 views

Isolation Forest - Cost function and optimization method

I have two questions about isolation forest. I may not understand how it works correctly but I just wonder: What is the cost function of the isolation forest? What is the optimization method to ...
Amhs_11's user avatar
  • 333
1 vote
0 answers
6k views

How to Tune Isolation Forest?

Many online blogs talk about using Isolation Forest for anomaly detection. But I got a very poor result. The data used is house prices data from Kaggle. I used IForest and KNN from pyod to identify 1% ...
etang's user avatar
  • 1,007
1 vote
1 answer
3k views

Anomaly Detection over multivariate categorical and numerical predictors

I am trying to implement Anomaly Detection over a multivariate dataset having categorical and numerical predictors. If we consider the below sample records, product_type, company_type and currency are ...
Dhaval Simaria's user avatar
0 votes
0 answers
261 views

Can you use the isolation forest algorithm on large sample sizes?

The original isolation forest paper states that the algorithm works best on small subsamples, but is it okay to use it on large sample sizes or are other anomaly detection algorithms better?
ddx's user avatar
  • 111
1 vote
1 answer
422 views

Can you use the isolation forest algorithm on a large sample size?

I've been using the scikit learn sklearn.ensemble.IsolationForest implementation of the isolation forest to detect anomalies in my datasets that range from 100s of ...
ddx's user avatar
  • 111
0 votes
0 answers
66 views

Separating anomalys using Isolation Forest, Is my approach correct?

I am a fairly new guy to ML and I am having some trouble choosing a algorithm to the job for me. My data set consists physical measurements where part of the samples were contaminated. In this case, ...
Back up's user avatar
2 votes
0 answers
62 views

Identify the parameter causing the anomaly in a multivariate dataset

I have a payment transaction dataset with a large number of predictor variables. I am trying to build a model for anomaly detection and I have evaluated various algorithms/approaches for the same like ...
Dhaval Simaria's user avatar
1 vote
1 answer
883 views

Meaning Of The Terms In Isolation Forest Anomaly Scoring

In an isolation forest the anomaly score of a point is given by: $$2^{\frac{-E(h(x))}{c(m)}}$$ Now supposedly c(m) is the average length to termination in the search tree. And, E(h(x)) is the ...
Trajan's user avatar
  • 449
0 votes
1 answer
353 views

Discordance between various methods of multivariate outliers detection

Here is a small "toy example" dataset, with 15 individuals described by 6 variables (this is R language): ...
Philopolis's user avatar
0 votes
0 answers
274 views

Oversampling/Undersampling in respect to Train and Test - Isolation Forest

I've got a quite imbalanced data set. 144.496 : 162 -> ratio of 1000:1 I would like to use IsolationForest to detect the 162 anomalys. I've already split the data. However, the iForest doesn't ...
Marvin's user avatar
  • 1
0 votes
0 answers
20 views

Model Examples always with 1 or 2 features

Why are all the model examples that I see on sklearn (e.g., https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.LocalOutlierFactor.html or https://scikit-learn.org/stable/auto_examples/...
theStud54's user avatar
  • 101
5 votes
1 answer
6k views

Isolation forest with categorical data?

I understand how isolation forests can work with numeric data, but I wonder how it can work with categorical data? Also, at least when working with Sci-kit-Learn, the recommendation I saw was to ...
Maverick Meerkat's user avatar
1 vote
1 answer
85 views

Isolation Forest Numerical Example

I'm looking for a proper numerical example to understand Isolation Forests Algorithm correctly. I've read the paper : https://github.com/mgckind/iso_forest/blob/master/icdm08b.pdf, but I want to ...
zubug55's user avatar
  • 111
6 votes
2 answers
2k views

Isolation Forest and average/expected depth formula

The Isolation Forest algorithm (Liu, Fei Tony, Kai Ming Ting, and Zhi-Hua Zhou. "Isolation forest." 2008 Eighth IEEE International Conference on Data Mining. IEEE, 2008. - link: https://cs.nju.edu.cn/...
anymous.asker's user avatar
2 votes
0 answers
606 views

Does the isolation forest care about integer-encoded categorical variables?

The isolation forest (initial paper, follow-up paper) as well as the proposed extended isolation forest (paper) seem like very appealing unsupervised anomaly detection techniques. However, the ...
robot_2077198's user avatar
7 votes
2 answers
10k views

How to get top features that contribute to anomalies in Isolation forest

I am using Isolation forest for anomaly detection on multidimensional data. The algorithm is detecting anomalous records with good accuracy. Apart from detecting anomalous records I also need to find ...
Amar's user avatar
  • 71
4 votes
2 answers
2k views

Strange encoding for categorical features

I am reading through https://arxiv.org/pdf/1609.06676.pdf which presents an extension of the isolation forest algorithm so that categorical features may be taken into account. On page 5, the authors ...
robot_2077198's user avatar
4 votes
2 answers
8k views

Unsupervised Learning: Train Test division

I have one conceptual question. In Unsupervised Learning, when I have no labels. The anomaly detection model (Isolation forests, Autoencoders, Distance-based methods etc.), it should fit on a ...
Manu Sharma's user avatar
12 votes
5 answers
19k views

Feature Importance in Isolation Forest

In an unsupervised setting for higher-dimensional data (e.g. 10 variables (numerical and categorical), 5000 samples, ratio of anomalies likely 1% or below but unknown) I am able to fit the isolation ...
robot_2077198's user avatar
4 votes
1 answer
2k views

Feature Importances for Isolation Forest?

Is there a way to dig in to sklearn's Isolation Forest algorithm to understand why data is scored as an inlier or outlier? I see that you can produce a score with decision_function (for data used to ...
Maripqz's user avatar
  • 41
4 votes
0 answers
3k views

Unsupervised anomaly detection - metric for tuning Isolation Forest parameters

I have a project, in which, one of the stages is to find and label anomalous data points, that are likely to be outliers. As a first step, I am using Isolation Forest algorithm, which, after plotting ...
Bonzogondo's user avatar
4 votes
1 answer
3k views

Validating Isolation Forests

I have a dataset where I'd like to perform anomaly detection with an Isolation Forest. I don't have any way to validate the model (my data is not labeled - that's why I'm using unsupervised learning) -...
lte__'s user avatar
  • 207
15 votes
3 answers
16k views

scikit-learn IsolationForest anomaly score

According to IsolationForest papers (refs are given in documentation) the score produced by Isolation Forest should be between 0 and 1. The implementation in scikit-learn negates the scores (so high ...
DAF's user avatar
  • 187
5 votes
0 answers
904 views

What data assumptions are made when using an isolation forest?

I am performing anomaly detection with a high-dimensional zero-inflated data-set. This rules out options like standard Gaussian anomaly detection as no amount of transformation can make my features "...
Darrrrrren's user avatar
1 vote
1 answer
6k views

Does Isolation Forest need an anomaly sample during training?

I am using Isolation Forest for anomaly detection (scikit implementation in python). My data have 1000 dimensions. My normal data, which I use for training Isolation Forest model, has only to features ...
user1872329's user avatar
0 votes
0 answers
422 views

Outlier Detection giving almost all training points as outliers

I am trying to run an outlier detection algorithm using python and I have used OneClassSVM and Isolation Forest on a dataset of about 30 points with 5 outliers. I have trained on the 25 points and ...
silent_dev's user avatar