Feature Selection in Isolation Forest? How to use kurtosis?

Question

I'm unsure about which is the best approach for feature selection for Isolation Forest. I am using a dataset which initially has 5 numerical columns and 50 categorical. After pre-processing and feature engineering I end up with 9 numerical columns and by using one-hot-encoding I end up with 70 binary columns. Based on a paper, including categorical attributes is not a good idea for IForest - but that is mostly dataset-specific. I'm reading the 2012 Isolation Forest paper (https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/tkdd11.pdf) and at some point the author says:

In this experiment, we study a special case of high dimensional data in which
data sets have a large number of irrelevant attributes. We show that iForest has a
significant advantage in processing time. We simulate these high dimensional data
sets using the first thirteen data sets introduced in Table III. For each data set,
uniformly distributed random attributes, valued between 0 and 1 are added. Such
that, there is a total of 512 attributes in each data set. We use a simple statistical
test, Kurtosis [Joanes and Gill 1998], to select an attribute subspace from the
sub-sample before constructing each iTree. Kurtosis measures the ‘peakness’ of
a univariate distribution. Kurtosis is sensitive to the presence of anomalies and
hence it is a good attribute selector for anomaly detection. After Kurtosis has
provided a ranking for each attribute, a subspace of attributes is selected according
to this ranking to construct each tree.

But it is not clear to me how they used Kurtosis for feature selection. Although I understand that they state they use it on the sub-samples, I tried it on my dataset as a whole. Hence, I used scipy's kurtosis for each column and then ranked their values. It seems that the quoted text suggests that they picked the highest valued kurtosis attributes. But I'm not sure about it. In my test, if I pick an arbitrary number of attributes in descending order then I simply get awful results. Whereas when I pick in ascending order I get quite good results. But this seems to contradict the author above. Furthermore, by using domain knowledge or just common sense the ones that have higher kurtosis seems completely useless.

Any clarifications are appreciated. Also, any other suggestion for feature selection.

Answer 1

Kurtosis measures outliers (anomalies), not "peakness" (and not "peakedness" either). So if your goal is to detect anomalies, then by ranking the variables from highest to lowest kurtosis, you will indeed pick the variables that exhibit the more extreme anomalies. Ranking in the opposite direction will isolate the more "vanilla" variables as regards anomalies.

Not being familiar with IForest, I can't comment on that.