Good F1 score for anomaly detection

Question

I will be running a machine learning anomaly detection algorithm on some data. It's a binary classification problem: each data point is either anomalous or normal. I have some known anomalous data that I can use in testing the features I select.

I will be using the F1 score for testing the features I've selected for this algorithm. What would be considered a good F1 score?

Answer 1

You may have guessed it already, but it depends on ...

1. what the data allows you to achieve at maximum. Regarding this, please see Expected best performance possible on a data set
2. the cost of a wrong prediction and the benefit of a detected anomaly. I recommend The Foundations of Cost-Sensitive Learning by Charles Elkan for creation of such a cost-benefit-matrix.

So in summary, as long as the F1-score is significantly better than a random classifier (or any other dummy approach) and the cost-benefit-calculation based upon the model allows the conclusion that it is useful in practice, the corresponding F1-score can be considered as good.

Edit: What is the F1-score of a random classifier ?

Let's say we have 2 classes $c_1,c_2$.

Let's denote the a priori class probabilties as $p(c_1),p(c_2)$, $p(c_1)+p(c_2)=1$, where $p(c_k)$=number of instances with class $c_k$ divdided by number of all instances.

If the random classifier is presented an instance, it selects a class with probablitiy $p_{random}(c_k)$. Two major options exist:

• $p_{random}(c_k)=p(c_k)$. This classifier will maximize the overall accuracy.
• $p_{random}(c_k)=\frac{1}{2}$. This classifier will maximize recall for a minor class (as it is in case of anomaly detection) for the cost that more instances of the major class are not caught (i.e. the recall of the major class decreases). That's why a cost-maxtrix is needed.

Since the random classifier assigns the class independently of the presented instance, the prior class probabilities and hence the precision remain the same. So the precision for class k is just $precision(c_k)=p(c_k)$

Using the same independence argument, Recall, which is ratio of items of class k which have been correctly classified, is just $p_{random}(c_k)$.

Then, the F1-score for class k is (reference)

$F_1(c_k)=2\frac{precision*recall}{precision + recall}$

=$2\frac{p(c_k)p_{random}(c_k)}{p(c_k)+p_{random}(c_k)}$

Example:

Let $p(anomaly)=0.01,p(\neg anomaly)=0.99$. Then

If $p_{random}(c_k)=p(c_k)$:

precision(anomaly)=0.01, recall(anomaly)=0.01, F1(anomaly)~0.01 (accuracy=0.9892)

If $p_{random}(c_k)=\frac{1}{2}$:

precision(anomaly)=0.01, recall(anomaly)=0.5, F1(anomaly)~0.01961 (accuracy=0.5)

I have "verified" the calculation of precision and recall using the following MC-simulation (sorry for the quality, my R is a little bit rusty).

size <- 10000
trials <- 10000

anomalyCount <- size*0.01
noAnomalyCount <- size-size*0.01
dat <- data.frame("id"=1:size,"class"=as.character(c(rep("anomaly",anomalyCount),rep("no_anomaly",noAnomalyCount))))

prandom <- 0.01

precisionValues <- rep(NA,trials)
recallValues <- rep(NA,trials)

for(i in 1:trials){
    predictions <- sample(c("anomaly","no_anomaly"),size,replace=T,prob=c(prandom,1-prandom))
    anomalyInds <- which(predictions =="anomaly")   

    hits <- sum(dat$class[anomalyInds] == predictions[anomalyInds])
    precisionValues[i] <- hits / length(anomalyInds)
    recallValues[i] <- hits/anomalyCount
}

which delivers for $p_{random}(anomaly)=0.5$

[1] "average precision 0.00998538701407172 with error +/- 1.00354527409206e-07"
[1] "average recall 0.499318 with error +/- 1.00354527409206e-07"

and for $p_{random}(anomaly)=p(anomaly)=0.01$

[1] "average precision 0.0100008820210962 with error +/- 9.88994082273052e-07"
[1] "average recall 0.009979 with error +/- 9.88994082273052e-07"