4
$\begingroup$

In an unsupervised setting for higher-dimensional data (e.g. 10 variables (numerical and categorical), 5000 samples, ratio of anomalies likely 1% or below but unknown) I am able to fit the isolation forest and retrieve computed anomaly scores (following the original paper and using the implementation in scikit-learn). This gives me a ranking of potential anomalies to consider. However, how would I further assess the validity of these flags? How can I understand which feature has contributed to the anomaly score the most? Feature importance techniques usually applied in random forests do not seem to work in case of the isolation forest.

Interested to hear your thoughts. Any help is very appreciated.

$\endgroup$
  • 1
    $\begingroup$ It's not clear to me that feature importance is even a meaningful concept for isolation forests. By definition, anomalies are any points abnormally far from most of the data. There are no "most important" features that determine an anomaly, as a point could be far in any direction. $\endgroup$ – user20160 Feb 20 '19 at 22:08
4
$\begingroup$

I believe it was not implemented in scikit-learn because in contrast with Random Forest algorithm, Isolation Forest feature to split at each node is selected at random. So it is not possible to have a notion of feature importance similar to RF.

Having said that, If you are very confident about the results of Isolation Forest classifier and you have a capacity to train another model then you could use the output of Isolation Forest i.e -1/1 values as target-class to train a Random Forest classifier. This will give you feature importance for detecting anomaly.

Please note that I haven't tried this myself, so I can't comment on accuracy of this proposed approach.

$\endgroup$
  • $\begingroup$ Interesting technique. Any reference where you have seen it used? $\endgroup$ – hipoglucido Feb 7 at 15:38
  • $\begingroup$ That's a good approach and its being used in many cases. It is similar to fitting a decision tree on cluster labels. In both cases you have to be sure that the model fit well (overfit actually). $\endgroup$ – Ilker Kurtulus Feb 14 at 8:47
2
$\begingroup$

Have you tried looking at SHAP statistics as a way of measuring feature importance for your isolation forest exercise? Here's a good explanation of SHAP https://towardsdatascience.com/explain-your-model-with-the-shap-values-bc36aac4de3d and you can build an explainer object with any tree based model. From there you can also look at how your features affect individual predictions.

$\endgroup$
  • $\begingroup$ As the existing answer notes, Isolation Forests' trees are not standard decision trees, so shap probably won't produce anything useful. $\endgroup$ – Ben Reiniger Feb 21 at 16:11
  • $\begingroup$ @BenReiniger I’m unsure what you mean Isolation Forests are not standard decision trees. Like a Random Forest, IF builds a large number of trees and like a Random Forest the Isolation Forest subsamples the feature space and subsequently splits on a random value for a given feature. It does this many times to produce uncorrelated trees and the associated error of a specific tree. Path lengths are measured then split off anomalous from non anomalous data. You can then see which features explain why individual data points are considered anomalous, which can also be aggregated across data points. $\endgroup$ – Curtis Feb 22 at 0:00
  • $\begingroup$ Hrm, you may be right: "explaining" the contribution to the path-length-score should work. (+1) I wonder whether the TreeExplainer will be able to do it, or if you'd need the generic KernelExplainer? (My original complaint was that everything about a split in an isolation forest is random: the feature as well as the splitting point. Then there's no information gain or impurity reduction, and there's no target at all, but shap shouldn't care about any of that.) $\endgroup$ – Ben Reiniger Feb 22 at 17:03
  • $\begingroup$ Ah, indeed, shap now supports IsolationForest: stats.stackexchange.com/a/451518/232706 , github.com/slundberg/shap/issues/237 , github.com/slundberg/shap/pull/784 $\endgroup$ – Ben Reiniger Feb 26 at 21:13
  • $\begingroup$ @BenReiniger yes I’ve used it previously and it definitely provides a ton of insight for these unsupervised modeling situations $\endgroup$ – Curtis Feb 26 at 23:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.