1
$\begingroup$

I'm unsure about which is the best approach for feature selection for Isolation Forest. I am using a dataset which initially has 5 numerical columns and 50 categorical. After pre-processing and feature engineering I end up with 9 numerical columns and by using one-hot-encoding I end up with 70 binary columns. Based on a paper, including categorical attributes is not a good idea for IForest - but that is mostly dataset-specific. I'm reading the 2012 Isolation Forest paper (https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/tkdd11.pdf) and at some point the author says:

In this experiment, we study a special case of high dimensional data in which
data sets have a large number of irrelevant attributes. We show that iForest has a
significant advantage in processing time. We simulate these high dimensional data
sets using the first thirteen data sets introduced in Table III. For each data set,
uniformly distributed random attributes, valued between 0 and 1 are added. Such
that, there is a total of 512 attributes in each data set. We use a simple statistical
test, Kurtosis [Joanes and Gill 1998], to select an attribute subspace from the
sub-sample before constructing each iTree. Kurtosis measures the ‘peakness’ of
a univariate distribution. Kurtosis is sensitive to the presence of anomalies and
hence it is a good attribute selector for anomaly detection. After Kurtosis has
provided a ranking for each attribute, a subspace of attributes is selected according
to this ranking to construct each tree.

But it is not clear to me how they used Kurtosis for feature selection. Although I understand that they state they use it on the sub-samples, I tried it on my dataset as a whole. Hence, I used scipy's kurtosis for each column and then ranked their values. It seems that the quoted text suggests that they picked the highest valued kurtosis attributes. But I'm not sure about it. In my test, if I pick an arbitrary number of attributes in descending order then I simply get awful results. Whereas when I pick in ascending order I get quite good results. But this seems to contradict the author above. Furthermore, by using domain knowledge or just common sense the ones that have higher kurtosis seems completely useless.

Any clarifications are appreciated. Also, any other suggestion for feature selection.

$\endgroup$
1
  • $\begingroup$ nice question and good investigation you did already. not sure why others did not upvote $\endgroup$ Jun 25, 2023 at 16:25

1 Answer 1

0
$\begingroup$

Kurtosis measures outliers (anomalies), not "peakness" (and not "peakedness" either). So if your goal is to detect anomalies, then by ranking the variables from highest to lowest kurtosis, you will indeed pick the variables that exhibit the more extreme anomalies. Ranking in the opposite direction will isolate the more "vanilla" variables as regards anomalies.

Not being familiar with IForest, I can't comment on that.

$\endgroup$
2
  • $\begingroup$ Are you intending to contradict the contents of the article quoted by the question? Please be clear on this since the question is logically following the procedures of the article. So you're not actually arguing about a misapplication of the article's steps by the OP but rather that the premise of the procedure is incorrect. Given you admit that you are unfamiliar with IForest then well you should likely do that homework first as part of preparing an answer? $\endgroup$ Jun 25, 2023 at 16:29
  • $\begingroup$ Anomaly detection is indeed a correct application of kurtosis. However, the statement that kurtosis measures 'peakness' [sic] is absolutely wrong. $\endgroup$ Jun 26, 2023 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.