I'm doing some anomaly detection, basically classifying stuff as normal (0) or aberrant (1)

As with all anomalies, they are rare, so during the train/test phase it's not good enough for me to just look at general training error (which would be reduced to 10% if I just guessed class 0 (normal) all the time!)

with 90% non-aberrant data, I've decided to use A/B as my metric for training accuracy

where A = number of false negatives (saying it's normal when it's actually aberrant)

and B = # of positives (both false, and real)

So now, when I make a prediction, I also output that value (actually the average of that value when I rerun the test 15 times, and give each run an unweighted vote). And then I can say "oh, my test value is small enough, I can be confident in the prediction"

My question is whether or not this is sound practice, and if there are any better metrics for anomaly detection?


Your problem is binary classification with imbalanced class weights in your train / test data. What people sometimes do is report F-score that weights recall more than precision (i.e. you want to be make sure that you find the anomalies but are more OK with some of your predictions being false positives). You can also formulate F-score based on Type I and Type II errors rather than recall and precision - it's the same formula.

Here is the formula for $F_{\beta}$ score:

enter image description here

You want to compute $F_2$ score, which will penalize false negatives.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.