0
$\begingroup$

I've got a quite imbalanced data set.

144.496 : 162 -> ratio of 1000:1

I would like to use IsolationForest to detect the 162 anomalys. I've already split the data.

However, the iForest doesn't work well with this ratio (contamination = 0.0015).

Thus, I also tried undersampling (SMOTE) on the train set by chosing a ratio of 1:1, whereas I changed the contamination to 0.5 .. as consequence, the False Positives are very high..

Therefore, I would like to know, how do I have to deal with the ratios in respect Train / Test? Previously my idea was to balance the data before training, but I have to specify the "contamination" value for the iForest, which is why the test results are catastrophic.

Could anybody provide me with ideas to tackle this issue?

Idea: Use Case

ConfusionMatrix of SMOTE]

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ Have you looked at the anomalous data? Can you see the difference to normal data? $\endgroup$ – jonnor Feb 11 at 21:07
  • $\begingroup$ Have you tried to remove the anomalous data from the training set and using 0% contamination. Would then use anomalies only for val/test sets $\endgroup$ – jonnor Feb 11 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.