Is anomaly detection only done using deep learning? Or can you use something like a random forest model to detect anomalies? For reference, I'm performing classification with a binary response. My dataset is fairly large (several million rows) with about 99.6% and 0.4% in each of the binary classes. I'm curious to see if there are ways to improve random forest to handle such cases.
3
-
$\begingroup$ Haven't come across this before, thank you, good to know! I'll be curious to see if the sparklyR package can implement this one. $\endgroup$– piper180Commented Mar 7, 2022 at 21:10
-
3$\begingroup$ Deep learning may or may not be useful depending on the problem, but it's certainly not required. There's a whole body of literature and techniques for anomaly detection that predate modern deep learning $\endgroup$– user20160Commented Mar 8, 2022 at 19:35
-
$\begingroup$ @user20160 thank you, this is what I needed. I will look into these methods for anomaly detection. $\endgroup$– piper180Commented Mar 14, 2022 at 19:55
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
isolation-forest is a method that uses randomized decision tress (similar to random forest) to identify outliers/anomalies in data sets.
"Isolation Forest" by Fei Tony Liu, Kai Ming Ting and Zhi-Hua Zhou.
Most existing model-based approaches to anomaly detection construct a profile of normal instances, then identify instances that do not conform to the normal profile as anomalies. This paper proposes a fundamentally different model-based method that explicitly isolates anomalies instead of profiles normal points. To our best knowledge, the concept of isolation has not been explored in current literature. The use of isolation enables the proposed method, iForest, to exploit sub-sampling to an extent that is not feasible in existing methods, creating an algorithm which has a linear time complexity with a low constant and a low memory requirement. Our empirical evaluation shows that iForest performs favourably to ORCA, a near-linear time complexity distance-based method, LOF and Random Forests in terms of AUC and processing time, and especially in large data sets. iForest also works well in high dimensional problems which have a large number of irrelevant attributes, and in situations where training set does not contain any anomalies
