0
$\begingroup$

I've got a quite imbalanced data set.

144.496 : 162 -> ratio of 1000:1

I would like to use IsolationForest to detect the 162 anomalys. I've already split the data.

However, the iForest doesn't work well with this ratio (contamination = 0.0015).

Thus, I also tried undersampling (SMOTE) on the train set by chosing a ratio of 1:1, whereas I changed the contamination to 0.5 .. as consequence, the False Positives are very high..

Therefore, I would like to know, how do I have to deal with the ratios in respect Train / Test? Previously my idea was to balance the data before training, but I have to specify the "contamination" value for the iForest, which is why the test results are catastrophic.

Could anybody provide me with ideas to tackle this issue?

Idea: Use Case

ConfusionMatrix of SMOTE]

Improve this question
$\endgroup$
2
  • $\begingroup$ Have you looked at the anomalous data? Can you see the difference to normal data? $\endgroup$
    – Jon Nordby
    Feb 11, 2020 at 21:07
  • $\begingroup$ Have you tried to remove the anomalous data from the training set and using 0% contamination. Would then use anomalies only for val/test sets $\endgroup$
    – Jon Nordby
    Feb 11, 2020 at 21:10

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.