I inspired by this notebook, and I'm experimenting IsolationForest algorithm for anomaly detection context on the SF version of KDDCUP99 dataset, including 4 attributes. The data is directly fetched from sklearn and after preprocessing (label encoding the categorical feature) passed to the IF algorithm with default setup (except for n_estimator, which I used GridsearchCV to find the optimum number of trees).

The problem is based on the KDDCUP99 dataset documentation, and its User guide and my calculation in the following implementation the anomaly rate is 0.5% and it means if I set contamination=0.005, it should give me the best results but surprisingly, it does not!

The full code is as follows:

from sklearn import datasets
from sklearn import preprocessing
from sklearn.model_selection import train_test_split

from sklearn.ensemble import IsolationForest
from sklearn.metrics import classification_report, ConfusionMatrixDisplay
from sklearn.metrics import confusion_matrix
from sklearn.metrics import recall_score, silhouette_score, roc_curve, roc_auc_score, f1_score, precision_recall_curve, auc
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer
from sklearn.metrics import accuracy_score

import pandas as pd
import numpy as np
import seaborn as sns
import itertools
import matplotlib.pyplot as plt
import datetime

%matplotlib inline

def byte_decoder(val):
    # decodes byte literals to strings
    return val.decode('utf-8')
def plot_confusion_matrix(cm, title, classes=["Normal", "Anomaly"],
                          cmap=plt.cm.Blues, save=False, saveas="MyFigure.png"):
    # print Confusion matrix with blue gradient colours
    cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=90)
    plt.yticks(tick_marks, classes)

    fmt = '.1%'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                color="white" if cm[i, j] > thresh else "black")

    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    if save:
        plt.savefig(saveas, dpi=100)

#Load Dataset KDDCUP99 from sklearn
target = 'target'
sf = datasets.fetch_kddcup99(subset='SF', percent10=False)
                  columns=["duration", "service", "src_bytes", "dst_bytes"])
assert len(dfSF)>0, "SF dataset no loaded."

anomaly_rateSF = 1.0 - len(dfSF.loc[dfSF[target]==b'normal.'])/len(dfSF)

"SF Anomaly Rate is:"+"{:.1%}".format(anomaly_rateSF)
#'SF Anomaly Rate is: 0.5%'

#Data Processing 
toDecodeSF = ['service']
# apply hot encoding to fields of type string
# convert all abnormal target types to a single anomaly class

dfSF['binary_target'] = [1 if x==b'normal.' else -1 for x in dfSF[target]]
leSF = preprocessing.LabelEncoder()

for f in toDecodeSF:
    dfSF[f + " (encoded)"] = list(map(byte_decoder, dfSF[f]))
    dfSF[f + " (encoded)"] = leSF.fit_transform(dfSF[f])

for f in toDecodeSF:
  dfSF.drop(f, axis=1, inplace=True)

dfSF.drop(target, axis=1, inplace=True)

#check rate of Anomaly for setting contamination parameter in IF
dfSF["binary_target"].value_counts() / np.sum(dfSF["binary_target"].value_counts())

#train & data split
X_train_sf, X_test_sf, y_train_sf, y_test_sf = train_test_split(dfSF.drop('binary_target', axis=1), dfSF['binary_target'], 
                                                                test_size=0.33, random_state=11, stratify=dfSF['binary_target'])

clfIF = IsolationForest(max_samples="auto", random_state=11, contamination = "auto", n_estimators=100, n_jobs=-1)

y_pred_train = clfIF.predict(X_train_sf)

#Results on SF training set:
print(classification_report(y_train_sf, y_pred_train, target_names=["Anomaly", "Normal"]))
print("AUC: ", "{:.1%}".format(roc_auc_score(y_train_sf, y_pred_train)))
#AUC is about 93%.

#Results on SF training set:
clfIF = IsolationForest(max_samples="auto", random_state=11, contamination = "auto", n_estimators=100, n_jobs=-1)

clfIF.fit(X_train_sf, y_train_sf)
y_pred_test = clfIF.predict(X_test_sf)
#AUC is about 93%.

scoring = {'AUC': 'roc_auc', 'Recall': make_scorer(recall_score, pos_label=-1)}
gs = GridSearchCV(IsolationForest(max_samples="auto", random_state=11, n_jobs=-1),
                  param_grid={'n_estimators': range(1, 110, 5)},
                  scoring=scoring, refit='Recall', return_train_score=True, cv=3, verbose=1, n_jobs=-1)
gs.fit(X_train_sf, y_train_sf)
results = gs.cv_results_
max_recall_n_estimators = pd.DataFrame(results).iloc[np.argmax(pd.DataFrame(results)["mean_test_Recall"])]["param_n_estimators"]

fig, ax = plt.subplots(figsize=(8, 20))
# fig.tight_layout()

for idx, cont in zip(range(1, 4), [0.005, 0.1, 0.2]):
    iso_for_ = IsolationForest(random_state=11, 
    iso_for_.fit(X_train_sf, y_train_sf)
    y_pred_ = iso_for_.predict(X_test_sf)

    ax = plt.subplot(1, 3, idx)
    cm = confusion_matrix(y_test_sf, y_pred_)
    cm = np.round(cm / cm.sum(axis=1)[:, np.newaxis], decimals=2)
    cmd = ConfusionMatrixDisplay(cm, display_labels=["Anomaly", "Normal"])
    cmd.plot(ax=ax, xticks_rotation=45, cmap=plt.cm.Blues, values_format=".2%")
    cmd.ax_.set_title(f"IF (contamination = {cont}) \nconfusion matrix - SF")

fig.text(0, 0.5, "True Label", rotation=90, va='center')
fig.text(0.4, 0.4, 'Predicted Label', ha='left')
plt.suptitle("Confusion matrix of sklearn IF for various contaminations (SF)", y=0.63)

This is the results of the confusion matrix: img

Probably I am missing something here, and any help will be highly appreciated.

Edit: We know that IF algorithm label anomaly labels based on outliers score results and try to plot them on train and test set for better realization as follows:

outlier_scores_train = iso_for_sf.decision_function(X_train_sf)
outlier_scores_test  = iso_for_sf.decision_function(X_test_sf)

train_df = pd.concat([X_train_sf.reset_index(drop=True), pd.DataFrame(outlier_scores_train, columns=["outlier score"])], axis=1)
test_df = pd.concat([X_test_sf.reset_index(drop=True), pd.DataFrame(outlier_scores_test, columns=["outlier score"])], axis=1)

sns.distplot(outlier_scores_train, kde=True, norm_hist=True, color="b", label="Trainset outlier scores")
sns.distplot(outlier_scores_test,  kde=True, norm_hist=True, color="r", label="Testset outlier scores", hist_kws={'alpha': 0.3})

plt.xlabel("outlier score")
plt.title("Outlier scores Distribution")



  • 2
    $\begingroup$ The isolation forest's findings are strongly influenced by all of the hyperparameters, so the choice to only tune n_estimators seems strange. It's possible that a different hyperparameter configuration would recover the desired contamination. Also, tuning the number of trees tends to be dominated by noise, especially when the number of trees is small, as it is here. And the tuning with a binary prediction is even more volatile. See: stats.stackexchange.com/questions/348245/… $\endgroup$
    – Sycorax
    Mar 16, 2021 at 15:08
  • $\begingroup$ +1 Thanks for your insightful input that could reason my unexpected results. Then I keep number of trees high around 100 and try to apply GridsearchCV on max_samples and contamination instead of ignoring them by default or auto and repeat my experiment. My aim was to optimize the hyperparameters using GridsearchCV and assign them to my final IF model. $\endgroup$
    – Mario
    Mar 16, 2021 at 15:19
  • $\begingroup$ I would also try bootstrapping, and tuning max_features to de-correlate the trees, as well as a much large number of trees (perhaps 1000). $\endgroup$
    – Sycorax
    Mar 16, 2021 at 15:22
  • $\begingroup$ Great idea. actually I was going to mention that but you mean I just try bootstrapping for max_feature? because I aapplied GridsearchCV for this parameter as well. but you are offering to bootstrap=True for training phase and not leaving it default max_feature=1.0 , right? In general how do you find this approach in this notebook? I'm expecting after this parametrization phase if I pass contamination =0.005 I get the best results in confusion matrix. $\endgroup$
    – Mario
    Mar 16, 2021 at 15:36

1 Answer 1


Contamination in isolation forest is basically a hyperparameter. It is used to pick the threshold for the score to distinguish anomalies from non-anomalies. It should be close to the proportion of anomalies assuming that your training data is perfectly representative to the test (real!) data and that the algorithm was able to learn the representation of the data reasonably well for this task. As you can see, those are quite a big assumptions to be met.

This may be more obvious with simpler algorithm like EllipticEnvelope, that fits a multivariate Gaussian to your data and cuts-off the values from low density regions based on the value of the contamination hyperparameter. It would work perfectly if you set the value of contamination to the proportion of anomalous samples if your data is distributed as a multivariate Gaussian, but gives no such guarantees for other kinds of data. Isolation forest is a bit more complicated, but same problems apply.

TL;DR Just treat it as a hyperparameter.

  • $\begingroup$ Thanks for your input. I'm approaching as you mentioned. If you consider the edit part of my post I plotted the anomaly score distribution on train and test dataset for better understanding. The question is considering kddcup99 dataset documentation says 0.5% anomaly or contamination for SF version, why I couldn't get the optimum results in its confusion matrix with low FP & FN? $\endgroup$
    – Mario
    Mar 16, 2021 at 15:11
  • 2
    $\begingroup$ @Mario for the same reason why any other machine learning algorithm does not give you a perfect fit. Anomaly detection algorithms learn the distribution of the data and then, based on the threshold, they cut-off a fraction of the least likely values given the distribution. So if you fit the wrong distribution and cut-off some values, then this gives you no guarantees whatsoever that those would be anomalies or that those would align with the proportion of the anomalies in the population. Since you always approximate the distribution, the threshold and the fit would never be exact. $\endgroup$
    – Tim
    Mar 16, 2021 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.