How to interpret Isolation Forest results on variations of train/test sets?

Question

I have a labelled dataset, originally intended for classification or clustering tasks, whose minority class is at 10%. I am investigating whether this problem can be tackled with anomaly detection methods. After preprocessing, I end up with 5669 instances, 288 of which are considered 'anomalies' (that's the minority class, around 5%). At first, I was troubled with what is the most appropriate setting to train IForest. I have tried:

1. train: all inliers + half of the outliers, test: remaining outliers

2. train: all inliers, test: all outliers

3. train: inliers only, test: consisting of 50% outliers, 50% inliers

In the first two cases, I get very accurate results (>88% accuracy, very accurate recall/precision metrics) for low max_samples (16) and contamination set to 'auto'. For case 1, if I set the max_samples to 256 then accuracy drops to 60% and if I use contamination equal to the class distribution (0.03) I get most of the anomalies classified as FN. In general, it seems there is a trend of higher max_samples -> shift from TP, FP to FN, TN (TP=correctly identified anomaly, TN=correctly identified normal point), which suggests that anomalies are masked.

I also tried:

4. train: inliers + all outliers (5111, 288), test on inliers (270) and I get 97% accuracy, 262 TN, 8 FP

5. train: inliers + all outliers (288, 288), test on remaining inliers and the predictions are split between TN and FP with 50% accuracy.

These figures suggest that IForest doesn't seem to be capable of picking up the anomalies accurately. However, although some of these variations were found in others' publications they do not match the real-world application of the model - as these variations do not simulate reality.

Instead, a paper suggests that for an offline setting IForest needs to be trained and scored on the same dataset whereas for an online setting a split train/test set needs to be used.

Subsequently, I experimented with:

6. train: all instances, test: all instances
7. train: 75% of data, test: 25% of data

In both cases, if I use contamination 'auto' I get 39% accuracy for max_samples=32 and 62% for max_samples=256 (whereas the same shift from TP,FP to TN,FN is observed). However, if I use the class distribution (0.05) as the contamination ratio, I get very high accuracy - but that's just the accuracy paradox as I get almost all instances classified as TN (e.g. TP: 31, FN: 257, FP: 253, TN: 5128 / TP: 8, FN: 61, FP: 73, TN: 1276 respectively for 6 & 7). In that case, playing with max_samples does not make any difference.

In a classification setting, RF and DT also return the accuracy paradox but if the minority class is oversampled then the classifiers are very accurate. However, in existing literature, some authors finding contradict each other (e.g. author1 tests x,y,z algorithm and reports bad results with y but author2 with similar or identical preprocessing reports y as highly accurate). Plotting my dataset in 2D does not show any anomaly clusters but rather anomalies are scattered between normal points.

Considering all the above my conclusion is that IForest is not suitable for this task. But I'm not sure how to explain it. Supposedly, IForest should be efficient vs masking and swamping and handle these types of anomalies. Is the answer simply the construct validity of the dataset?

Any opinions or pointers are greatly appreciated,

Thanks

Answer 1

Choosing a good decision threshold is absolutely critical to getting reasonable performance. This holds for any binary decision problem, but especially so with class imbalance and outlier/anomaly detection ("one-class" modelling). The "contamination" parameter in scikit-learn is one way of tuning this threshold that, and should generally be tuned as a hyperparameter, using a separate (labeled) validation set.

It is entirely expected that the best performance will be had when contamination matches the ratio of anomalies in the data.

To see the effect of different decision thresholds, use score_samples() to get continuous anomaly scores, and plot a Precision/Recall curve for the different decision thresholds.